A Novel Rule Refinement Method for SMT through Simulated ...tcci.ccf.org.cn/conference/2014/ppts/nlpcc/ppt126.pdfA Novel Rule Refinement Method for SMT through Simulated Post-Editing

A Novel Rule Refinement Method for

SMT through Simulated Post-Editing

Sitong Yang1,2, Heng Yu?1, and Qun Liu1,3

1. Key Laboratory of Intelligent Information Processing. Institute of Computing Technology, Chinese Academy of Sciences

2. University of Chinese Academy of Sciences

3. CNGL, School of Computing, Dublin City University

2014/12/23

2014/12/23

Post-Editing

Pros & Cons

Our method

Data set & Experiment

Conclusion & Furture Work

Post Editing(PE)

2014/12/23 Sitong Yang, Heng Yu, and Qun Liu. A Novel Rule Refinement Method for SMTthrough Simulated Post-Editing

>> Post Editing >> Pros & Cons >> Our method >> Data set & Experiment >> Conclusion & Furture Work

Post Editing(PE)



Automatic post editing

Post Editing(PE)

2014/12/23

MT

Sitong Yang, Heng Yu, and Qun Liu. A Novel Rule Refinement Method for SMTthrough Simulated Post-Editing


Post Editing(PE)

2014/12/23

MT Post Editing (SMT)



Post Editing(PE)

2014/12/23


Result



Post Editing(PE)

2014/12/23


Result

Multiple stream



Post Editing(PE)

2014/12/23

MT



Post Editing(PE)

2014/12/23

Post Editing MT



Post Editing(PE)

2014/12/23

Better MT Post Editing



Post Editing(PE)

2014/12/23

Single stream

Better MT Post Editing



2014/12/23

Post-Editing

Pros & Cons

Our method



Pros & Cons

2014/12/23

Pros:



Pros & Cons

2014/12/23

Pros:

• Better adaptation



Pros & Cons

2014/12/23

Pros:


• No additional burden for SMT



Pros & Cons

2014/12/23

Pros:



Cons:



Pros & Cons

2014/12/23

Pros:



Cons:

• Expensive



Pros & Cons

2014/12/23

Pros:



Cons:

• Expensive

• Hard to learn



2014/12/23

Post-Editing

Pros & Cons

Our method



Our method

2014/12/23

We Learn from PE results to enhance the original SMT Model.



Our method

2014/12/23


• Simulated Post Editing



Our method

2014/12/23



• Error-Driven Frame work



Our method

2014/12/23



• Error-Driven Frame work

Error Detection Rule Extraction Rule Filteration



Simulated PE

2014/12/23

Daniel [2010] formulated the task of simulated post-editing, wherein pregenerated reference translations are used as a stand-in for actual post-editing.



Simulated PE

2014/12/23

Machine Translation



Simulated PE

2014/12/23

Machine Translation

Human Post Editing



Simulated PE

2014/12/23

Machine Translation

Human Post Editing

PE



Simulated PE

2014/12/23

Machine Translation

Human Post Editing

PE



Expensive

Simulated PE

2014/12/23

Machine Translation

Human Post Editing

Reference

PE



Simulated PE

2014/12/23

Machine Translation

Human Post Editing

Reference

PE

SiPE



Simulated PE

2014/12/23

Machine Translation

Human Post Editing

Reference

PE

SiPE

Cheap



Error-Driven Rule Refinement




2014/12/23

This man lived a dog ’s life

这个人生活潦倒

Src:

Tgt:




2014/12/23



Src:

Tgt:




2014/12/23



Src:

Tgt:

Alignment Error!




2014/12/23


这个人生活一只狗的生活


Src:

Tgt:

MT:zhege ren shenghuo yizhi gou de shenghuo




2014/12/23




Src:

Tgt:

MT:

这个人生活潦倒Ref:zhege ren shenghuo liaodao

zhege ren shenghuo yizhi gou de shenghuo




2014/12/23




Src:

Tgt:

MT:





Error Detection



Error Detection



Editing distance

Error Detection



TERplus

Synonym match Stem match

Phrase substitution Shift

Deletion Word substitution

Insertion

Editing distance→

Error Detection



Error Detection



0

1000

2000

3000

4000

5000

6000

7000

8000

9000

Y T Ps S D Ws I Y T Ps S D Ws I 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Y: Synonym match

T: Stem match

Ps: Phrase subst i t ut i on

S: Shif t

D: Delet i on

Ws: Word subst i t ut i on

I: Insert i on

SiPE Distribution SiPE Precision

Error Detection



0

1000

2000

3000

4000

5000

6000

7000

8000

9000

Y T Ps S D Ws I Y T Ps S D Ws I

small sample hard to learn

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Y: Synonym match

T: Stem match


S: Shif t

D: Delet i on


I: Insert i on


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Y: Synonym match

T: Stem match


S: Shif t

D: Delet i on


I: Insert i on

Error Detection



0

1000

2000

3000

4000

5000

6000

7000

8000

9000

Y T Ps S D Ws I Y T Ps S D Ws I


Rule extration and Filteration

2014/12/23

Filteration




2014/12/23

Filteration • C (words of Context )

• P (words of Source side Substitution part)






这个人生活一只狗的生活MT:



C=1 P=4

Extration Monolingual rule：

生活一只狗的生活 ||| 生活潦倒




这个人生活一只狗的生活MT:






2014/12/23





Src:

MT:






2014/12/23






2014/12/23



Original Bilingual rule ：

lived a dog ‘s life ||| 生活一只狗的生活 ||| 0.5 0.0149508 0.4 7.97148e-06 2.718




2014/12/23



Original Bilingual rule ：

lived a dog ‘s life ||| 生活一只狗的生活 ||| 0.5 0.0149508 0.4 7.97148e-06 2.718

New rule:

lived a dog ‘s life |||生活潦倒 ||| 0.5 0.0149508 0.4 7.97148e-06 2.718



Filtering Criterion



Filtering Criterion



calming the emotions of

calming the feelings of

C=2 P=2

the beginning of the new

the opening

C=1 P=3

C=2 P=1

between the faculty members and

between teachers and

C=2 P=3

to open up and

opening up and

Filtering Criterion





C=2 P=2


the opening

C=1 P=3

C=2 P=1



C=2 P=3

to open up and

opening up and

Filtering Criterion





C=2 P=2


the opening

C=1 P=3

C=2 P=1



C=2 P=3

to open up and

opening up and

Filtering Criterion

2014/12/23

• Should Contain More Context ( c>=2 )

• More Accurated Substitution ( 2<=p<=5 )



2014/12/23

Post-Editing

Pros & Cons

Our method



Experiments-Setup

2014/12/23

• Baseline system: Moses:a state-of-art phrase-based SMT system

Hiero: Hierarchical phrase-based system

• Word-alignment tool: GIZA++

• Language model: SRILM toolkit

• MT evaluation metric: Case-insensitive Bleu-4, Ter-plus



Experiments-Setup

2014/12/23

• Data Set:

• SiPE dataset:

Training-set: 10-fold cross validation

1 10 9 8 7 6 5 4 3 2

SMT MT Result

10 9 8 7 6 5 4 3 2 1 10 group



Domain Language Training-set Dev-set Test-set

News C2E 240k Nist02 Nist04 Nist05 Nist06

Medical C2E 560K 1000 1000

Main Results:Rule Refinement Method

2014/12/23

• new

• medical



System Bleu TERp

04 05 06 avg 04 05 06 avg

moses 32.02 29.00 27.18 29.34 61.47 64.04 66.45 64.47

balanced 33.47 30.02 28.80 30.76 60.24 59.37 63.89 61.77

hiero 34.10 29.89 28.78 30.92 59.55 62.73 64.84 62.37

balanced 34.09 29.87 28.81 30.92 59.56 62.75 64.85 62.38

System Bleu TERp

moses 29.64 66.06

ours 30.15 63.80

hiero 29.48 63.53

ours 30.26 62.57

Main Results: Rule Filteration



BLEU Score

Rule-size(1000)

2014/12/23

Post-Editing

Pros & Cons

Our method



Conclusion & Furture work

2014/12/23

Conclusion: • a novel rule refinement method for SMT.

• a simulated post-editing paradigm to efficiently collect the training data.

• TER-Plus for translation error detection.

• a simple and effectively heuristic algorithm for rule-filteration.

• both phrase-based and syntax-based SMT systems.

• gains an overall improvement of 1.4 BLEU point without using any additional resources.



Conclusion & Furture work

2014/12/23

Furture work:

• test our method on more complex translation models.

• produce more powerful feedbacks to improve SMT systems.



Thanks for your attention!

2014/12/23

A Novel Rule Refinement Method for SMT through Simulated ...tcci.ccf.org.cn/conference/2014/ppts/nlpcc/ppt126.pdfA Novel Rule Refinement Method for SMT through Simulated Post-Editing

Documents