Stat-XFER: A General Framework for Search-based Syntax-driven MT Alon Lavie Language Technologies Institute Carnegie Mellon University Joint work with: Erik Peterson, Alok Parlikar, Vamshi Ambati, Abhaya Agarwal, Greg Hanneman, Kevin Gimpel, Edmund Huber
41
Embed
Stat-XFER: A General Framework for Search-based Syntax-driven MT
Stat-XFER: A General Framework for Search-based Syntax-driven MT. Alon Lavie Language Technologies Institute Carnegie Mellon University Joint work with: Erik Peterson, Alok Parlikar, Vamshi Ambati, Abhaya Agarwal, Greg Hanneman, Kevin Gimpel, Edmund Huber. Outline. Context and Rationale - PowerPoint PPT Presentation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Stat-XFER: A General Framework for
Search-based Syntax-driven MT
Alon LavieLanguage Technologies Institute
Carnegie Mellon University
Joint work with:Erik Peterson, Alok Parlikar, Vamshi Ambati, Abhaya Agarwal, Greg Hanneman, Kevin Gimpel, Edmund Huber
March 28, 2008 NIST MT-08: CMU Stat-XFER 2
Outline• Context and Rationale• CMU Statistical Transfer MT Framework• Automatic Acquisition of Syntax-based MT
Resources• Chinese-to-English System• Urdu-to-English System• Open Research Challenges• Conclusions
March 28, 2008 NIST MT-08: CMU Stat-XFER 3
Rule-based vs. Statistical MT• Traditional Rule-based MT:
– Expressive and linguistically-rich formalisms capable of describing complex mappings between the two languages
– Accurate “clean” resources– Everything constructed manually by experts– Main challenge: obtaining broad coverage
• Phrase-based Statistical MT:– Learn word and phrase correspondences automatically
from large volumes of parallel data– Search-based “decoding” framework:
• Models propose many alternative translations• Effective search algorithms find the “best” translation
– Main challenge: obtaining high translation accuracy
Research Goals• Long-term research agenda (since 2000) focused on
developing a unified framework for MT that addresses the core fundamental weaknesses of previous approaches:– Representation – explore richer formalisms that can capture
complex divergences between languages– Ability to handle morphologically complex languages– Methods for automatically acquiring MT resources from available
data and combining them with manual resources– Ability to address both rich and poor resource scenarios
• Focus has been on low-resource scenarios, scaling up to resource-rich scenarios in the past year
• Main research funding sources: NSF (AVENUE and LETRAS projects) and DARPA (GALE)
March 28, 2008 4NIST MT-08: CMU Stat-XFER
March 28, 2008 NIST MT-08: CMU Stat-XFER 5
CMU Statistical Transfer (Stat-XFER) MT Approach
• Integrate the major strengths of rule-based and statistical MT within a common framework:– Linguistically rich formalism that can express complex and
abstract compositional transfer rules– Rules can be written by human experts and also acquired
automatically from data– Easy integration of morphological analyzers and
generators– Word and syntactic-phrase correspondences can be
automatically acquired from parallel text– Search-based decoding from statistical MT adapted to find
the best translation within the search space: multi-feature scoring, beam-search, parameter optimization, etc.
– Framework suitable for both resource-rich and resource-poor language scenarios
The Lattice Decoder• Simple Stack Decoder, similar in principle to simple
Statistical MT decoders• Searches for best-scoring path of non-overlapping
lattice arcs• No reordering during decoding• Scoring based on log-linear combination of scoring
features, with weights trained using Minimum Error Rate Training (MERT)
• Scoring components:– Statistical Language Model– Rule Scores (currently: freq-based relative likelihood)– Lexical Probabilities– Fragmentation: how many arcs to cover the entire
translation?– Length Penalty: how far from expected target length?
March 28, 2008 NIST MT-08: CMU Stat-XFER 15
XFER Lattice Decoder0 0 ON THE FOURTH DAY THE LION ATE THE RABBIT TO A MORNING MEALOverall: -8.18323, Prob: -94.382, Rules: 0, Frag: 0.153846, Length: 0,
Words: 13,13235 < 0 8 -19.7602: B H IWM RBI&I (PP,0 (PREP,3 'ON')(NP,2 (LITERAL 'THE')
(NP2,0 (NP1,1 (ADJ,2 (QUANT,0 'FOURTH'))(NP1,0 (NP0,1 (N,6 'DAY')))))))>918 < 8 14 -46.2973: H ARIH AKL AT H $PN (S,2 (NP,2 (LITERAL 'THE') (NP2,0
• In progress or planned:– Arabic-to-English– Brazilian Portuguese-to-English– Native-Brazilian languages to Brazilian Portuguese– Hebrew-to-Arabic– Quechua-to-Spanish– Turkish-to-English
MT Resource Acquisition in Resource-rich Scenarios
• Scenario: Significant amounts of parallel-text at sentence-level are available– Parallel sentences can be word-aligned and parsed (at
least on one side, ideally on both sides)• Goal: Acquire syntax-based broad-coverage translation
lexicons and transfer rule grammars automatically from the data
• Syntax-based translation lexicons:– Broad-coverage constituent-level translation equivalents at
all levels of granularity– Can serve as the elementary building blocks for transfer
trees constructed at runtime using the transfer rules
March 28, 2008 17NIST MT-08: CMU Stat-XFER
Acquisition Process• Automatic Process for Extracting Syntax-driven Rules
and Lexicons from sentence-parallel data:1. Word-align the parallel corpus (GIZA++)2. Parse the sentences independently for both languages3. Run our new PFA Constituent Aligner over the parsed
sentence pairs4. Extract all aligned constituents from the parallel trees5. Extract all derived synchronous transfer rules from
the constituent-aligned parallel trees6. Construct a “data-base” of all extracted parallel
constituents and synchronous rules with their frequencies and model them statistically (assign them relative-likelihood probabilities)
March 28, 2008 18NIST MT-08: CMU Stat-XFER
PFA Constituent Node Aligner• Input: a bilingual pair of parsed and word-aligned
sentences• Goal: find all sub-sentential constituent alignments
between the two trees which are translation equivalents of each other
• Equivalence Constraint: a pair of constituents <S,T> are considered translation equivalents if:– All words in yield of <S> are aligned only to words in yield of <T>
(and vice-versa)– If <S> has a sub-constituent <S1> that is aligned to <T1>, then
<T1> must be a sub-constituent of <T> (and vice-versa) • Algorithm is a bottom-up process starting from word-
level, marking nodes that satisfy the constraints
March 28, 2008 19NIST MT-08: CMU Stat-XFER
PFA Node Alignment Algorithm Example
• Words don’t have to align one-to-one
• Constituent labels can be different in each language
• Tree Structures can be highly divergent
March 28, 2008 20NIST MT-08: CMU Stat-XFER
PFA Node Alignment Algorithm Example
• Aligner uses a clever arithmetic manipulation to enforce equivalence constraints
• Resulting aligned nodes are highlighted in figure
March 28, 2008 21NIST MT-08: CMU Stat-XFER
PFA Node Alignment Algorithm Example
Extraction of Phrases:• Get the Yields of the
aligned nodes and add them to a phrase table tagged with syntactic categories on both source and target sides
• Example:NP # NP :: 澳洲 # Australia
March 28, 2008 22NIST MT-08: CMU Stat-XFER
PFA Node Alignment Algorithm Example
All Phrases from this tree pair:
1. IP # S :: 澳洲 是 与 北韩 有 邦交 的 少数 国家 之一 。 # Australia is one of the few countries that have diplomatic relations with North Korea .
2. VP # VP :: 是 与 北韩 有 邦交 的 少数 国家 之一 # is one of the few countries that have diplomatic relations with North Korea
3. NP # NP :: 与 北韩 有 邦交 的 少数 国家 之一 # one of the few countries that have diplomatic relations with North Korea
4. VP # VP :: 与 北韩 有 邦交 # have diplomatic relations with North Korea5. NP # NP :: 邦交 # diplomatic relations6. NP # NP :: 北韩 # North Korea7. NP # NP :: 澳洲 # Australia
• Created a “Gold Standard” constituent alignments using the manual word-alignments and treebank trees– Node Alignments: 39874 (About 12/tree pair)– NP to NP Alignments: 5427
• Manual inspection confirmed that the constituent alignments are quite accurate (P > 0.80, R > 0.70)
• Evaluation: Run PFA Aligner with automatic word alignments on same data and compare with the “gold Standard” alignments
NP::NP [ “ 北韩” ] -> [“North” “Korea”](;Many to one alignment is a phrase) 28
Chinese-English System• Developed over past year under DARPA/GALE
funding (within IBM-led “Rosetta” team)• Participated in recent NIST MT-08 Evaluation• Large-scale broad-coverage system• Integrates large manual resources with
automatically extracted resources• Current performance-level is still inferior to
state-of-the-art phrase-based systems
March 28, 2008 29NIST MT-08: CMU Stat-XFER
Chinese-English System• Lexical Resources:
– Manual Lexicons (base forms): • LDC, ADSO, Wiki• Total number of entries: 1.07 million
– Automatically acquired from parallel data:• Approx 5 million sentences LDC/GALE data• Filtered down to phrases < 10 words in length• Full formed• Total number of entries: 2.67 million
March 28, 2008 30NIST MT-08: CMU Stat-XFER
Chinese-English System• Transfer Rules:
– 61 manually developed transfer rules– High-accuracy rules extracted from manually word-
aligned parallel dataCorpus Size (sens) Rules with
StructureRules (count>=2)
Complete Lexical rules
Parallel Treebank (3K) 3,343 45,266 1,962 11,521
993 sentences 993 12,661 331 2,199
Parallel Treebank (7K) 6,541 41,998 1,756 16,081
Merged Corpus set 10K 94,117 3160 29,340
March 28, 2008 31NIST MT-08: CMU Stat-XFER
March 28, 2008 NIST MT-08: CMU Stat-XFER
32
Translation Example• SrcSent 3 澳洲是与北韩有邦交的少数国家之一。• Gloss: Australia is with north korea have diplomatic relations DE few country world• Reference: Australia is one of the few countries that have diplomatic relations
with North Korea.
• Translation: Australia is one of the few countries that has diplomatic relations with north korea .
Urdu-to-English System• Primary condition did not allow parsing of parallel data
low-resource scenario:– Lexical resources: used provided LDC lexicon (tagged with POS),
plus lexical entries acquired from word-aligning parallel-data– XFER rules: manually developed (48 rules)– Language Model built from English side of parallel data
• Primary and Contrastive systems:– Cont-1: Built a phrase-based system using Moses– Primary: Used our multi-engine MT system to combine our XFER
system with our Moses system– Cont-2: Our constrained Stat-XFER system only– Cont-3: Unconstrained version of Stat-XFER with just a large LM– Cont-4: MEMT of Stat-XFER (Cont-2) and Columbia’s constrained
phrase-based system– Cont-5: similar to Cont-4, but with unconstrained systems
March 28, 2008 NIST MT-08: CMU Stat-XFER 36
Urdu-to-English System• Our official reported scores are incorrect due
to a “one-off” bug half way through our output• Corrected scores, as reported by NIST
March 28, 2008 NIST MT-08: CMU Stat-XFER 37
IBM BLEU METEORPrimary: MEMT (XFER+Moses) 0.1500 0.5087Cont-1: Moses only 0.1820 0.5069Cont-2: Stat-XFER only 0.1158 0.4528Cont-3: Stat-XFER unconstrained 0.1443 0.4740Cont-4: MEMT (XFER+Columbia) 0.1526 0.5142Cont-5: MEMT (XFER+CU unconst) 0.1623 0.5127
Open Research Questions• Our large-scale Chinese-English system is still
significantly behind phrase-based SMT. Why?– Feature set is not sufficiently discriminant?– Problems with the parsers for the two sides?– Weaker decoder?– Syntactic constituents don’t provide sufficient coverage?– Bugs and deficiencies in the underlying algorithms?
• The ISI experience indicates that it may take a couple of years to catch up with and surpass the phrase-based systems
• Significant engineering issues to improve speed and efficient runtime processing and improved search
March 28, 2008 38NIST MT-08: CMU Stat-XFER
Open Research Questions• Immediate Research Issues:
– Rule Learning:• Study effects of learning rules from manually vs
automatically word aligned data• Study effects of parser accuracy on learned rules• Effective discriminant methods for modeling rule scores• Rule filtering strategies
– Syntax-based LMs: • Our translations come out with a syntax-tree attached
to them• Add a syntax-based LM feature that can discriminate
between good and bad trees
March 28, 2008 39NIST MT-08: CMU Stat-XFER
Conclusions• Stat-XFER is a promising general MT framework, suitable
to a variety of MT scenarios and languages• Provides a complete solution for building end-to-end MT
systems from parallel data, akin to phrase-based SMT systems (training, tuning, runtime system)
• No open-source publicly available toolkits (yet), but we welcome further collaboration activities
• Complex but highly interesting set of open research issues