1 Neural Reranking Improves Subjective Quality of Machine Translation Neural Reranking Improves Subjective Quality of Machine Translation: NAIST at WAT 2015 Graham Neubig, Makoto Morishita, ○Satoshi Nakamura Nara Institute of Science and Technology (NAIST) 2015-10-16
20
Embed
Neural Reranking Improves Subjective Quality of Machine ......4 Neural Reranking Improves Subjective Quality of Machine Translation Reranking with Neural MT Models he has a cold Input
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Neural Reranking Improves Subjective Quality of Machine Translation
Neural Reranking Improves SubjectiveQuality of Machine Translation:
NAIST at WAT 2015
Graham Neubig, Makoto Morishita, ○Satoshi NakamuraNara Institute of Science and Technology (NAIST)
2015-10-16
2
Neural Reranking Improves Subjective Quality of Machine Translation
Statistical Translation FrameworksSymbolic Models
Phrase-based MT [Koehn+ 03]
Tree-to-String MT [Liu+ 06]
Encoder-Decoder [Sutskever+ 14]
Attentional [Bahdanau+ 15]
he has a cold
彼 は 風邪 を 引いている
he彼 は
has引いている
a cold風邪 を
he彼 は
has引いている
a cold風邪 を
彼 は 風邪
he has a cold
PRP VBZ DET NN
VP
NP
S
引いているを
Continuous-space (Neural) Models
he has a cold <s>
彼
彼
は
は
風邪
風邪
を
引いているを
<s>引いている
he has a cold
g1,...,g
4
a1
a2
a3
a4
hi-1
hi
ri-1
P(ei|F,e
1,...,e
i-1)
3
Neural Reranking Improves Subjective Quality of Machine Translation
Relative Merits/Demerits
● Symbolic Models✔ Inner workings well understood✔ Better at translating low-frequency words
● Continuous-space Models✔ Easier to implement✔ Produce more fluent output✔ Probabilistic model – can score output of other systems!
4
Neural Reranking Improves Subjective Quality of Machine Translation
Reranking with Neural MT Models
he hasa cold
Input
T2S/PBMT
N-best w/MT Features
1. 彼は寒さを持っている t=-0.5 l=-5.6 | -6.1
2. 彼は風邪を持っている t=-0.9 l=-5.8 | -6.7
3. 彼は風邪を引いた t=-1.5 l=-5.3 | -6.8
4. 彼は風邪がある t=-1.9 l=-5.4 | -7.3
NeuralModel
Neural Features
nmt=-5.8
nmt=-5.5
nmt=-3.4
nmt=-5.2
2. 彼は寒さを持っている t=-0.5 l=-5.6 nmt=-5.8 | -10.9
3. 彼は風邪を持っている t=-0.9 l=-5.8 nmt=-5.5 | -11.2
1. 彼は風邪を引いた t=-1.5 l=-5.3 nmt=-3.4 | -9.2
4. 彼は風邪がある t=-1.9 l=-5.4 nmt=-5.2 | -12.5
Rescored/Reranked N-best
Reranking
5
Neural Reranking Improves Subjective Quality of Machine Translation
What Do We Know About Reranking?
● Reranking greatly improves BLEU score, even overstrong baseline systems:
Sutskever+ 2014 Alkhouli+ 2015
en-frBLEU
Base 33.3Rerank 36.5
de-enBLEU
ar-enBLEU
Baseline 30.6 26.4Reranked 32.3 27.0
6
Neural Reranking Improves Subjective Quality of Machine Translation
What Don't We Know About Reranking?
● Does reranking improve subjective impressions ofresults?
● What are the qualitative differences before/after reranking with neural MT models?
7
Neural Reranking Improves Subjective Quality of Machine Translation
Experiments
8
Neural Reranking Improves Subjective Quality of Machine Translation
Experimental Setup
● Data: ASPEC Scientific Abstracts● Japanese ↔ English, Chinese
● Baseline: NAIST WAT2014 Tree-to-String System● Strong baseline achieving high scores● Implemented using Travatar (http://phontron.com/travatar)
● Neural MT Model: Attentional model● Trained ~500k sent., 256 hidden nodes, 2 model ensemble● Use words occurring 3+ times (vocab 50,000~80,000)● Trained w/ lamtram (http://github.com/neubig/lamtram)
● Automatic Evaluation: BLEU, RIBES
● Manual Evaluation: WAT 2015 HUMAN Score
9
Neural Reranking Improves Subjective Quality of Machine Translation
Results
en-ja ja-en zh-ja ja-zh0
10
20
30
40
50
BLE
U
en-ja ja-en zh-ja ja-zh70
75
80
85
90
Base
Rerank
RIB
ES
+1.6
+2.8
+2.5
+1.5 +1.8
+2.7
+1.4+1.8
Confirm what we know: Neural reranking helps automatic evaluation.
en-ja ja-en zh-ja ja-zh0
10203040506070
Base
Rerank
HU
MA
N
+12.5
+23.7 +10.0
+4.2
Show what we didn't know: Also help manual evaluation.
10
Neural Reranking Improves Subjective Quality of Machine Translation
What is Getting Better?
● Perform detailed categorization of the changes inJapanese-English results:
1. Is the sentence better/worse after ranking?
2. What is the main error corrected: insertion, deletion,substitution, reordering, or conjugation?
3. What is the detailed subcategory?
11
Neural Reranking Improves Subjective Quality of Machine Translation
Main Types of Errors Corrected/Caused
Type Improved Degraded % Impr.
Reordering 55 9 86%
Deletion 20 10 67%
Insertion 19 2 90%
Substitution 15 11 58%
Conjugation 8 1 89%
Total 117 33 78%
Overall improvements re-confirmed
In particular fixing reordering, insertion, andconjugation errors
12
Neural Reranking Improves Subjective Quality of Machine Translation