Machine translation (I) MT overview Ling 571 Fei Xia Week 9: 11/22/05 – 11/29/05.

Machine translation (I)MT overview

Ling 571

Fei Xia

Week 9: 11/22/05 – 11/29/05

Plan for the rest of the quarter

• MT:– Part I: MT overview: 11/22 -- 11/29– Part II: Word-based SMT: 12/1-12/6– Next quarter: seminar on MT

• Starting with a real baseline system• Improving the system in various ways• Reading and presenting recent papers

• Project presentation: 12/8

Homework and Quizzes

• Hw #8: due on 11/23 (tomorrow)

• Hw #9: due on 12/6

• Hw #10: presentation due on 12/8, report due on 12/13

• Quiz #3: 11/29

• Quiz #4: 12/6

Outline

• MT in a nutshell

• Major challenges in MT

• Major approaches

• Evaluation of MT systems

MT in a nutshell

Q1: what is the ultimate goal of translation?

• Translation: source language target language (ST)

• Ultimate goal: find the “perfect” translation for text in S, thus allowing people to “appreciate” the text in S without knowing S:– Accuracy: faithful to S, including meaning,

connotation, style, …– Fluency: the translation is as natural as an utterance

in T.

Q2: what are the perfect translations?

What do “Accuracy” and “Fluency” mean?

• Ex1: Complement / downplayer(1) A: Your daughter was phenomenal.(2) B: No. She was just so-so.

• Ex2: Greeting: how’s everything?– Old days: chi1 le5 ma5? (Have {you} eaten?)– 1980s -- now: fa1 le5 ma5? (Have (you) gotten rich?)– 2000s -- now: li2 le5 ma5? (Have (you) gotten divorced?)

• The answer: it depends

Q3: Can human always get the perfect translations?

• Novels: Shakespeare, Cao Xueqin, … - hidden messages: c1c0, c2 c0, c3 c0, c4 c0 c1’ c2’ c3’ c4’ • Word play, jokes, puns:

– What do prisoners use to call each other? – Cell phones.

• Concept gaps: double jeopardy, go Greek, fen sui, bei fen, ….

• Other constraints: lyrics, dubbing, poem.

“Crazy English” by Richard Lederer

• Let’s face it: English is a crazy language. There is no egg in eggplant or ham in hamburger, neither apple nor pine in pineapple.

• When a house burns up, it burns down. You fill in a form by filling it out and an alarm clock goes off by going on.

• When the stars are out, they are visible, but when the lights are out, they are invisible. And why, when I wind up my watch, I start it, but when I wind up this essay, I end it?

How to translate it?

• “Compound” words: Let’s face it: English is a crazy language. There is no egg in eggplant or ham in hamburger, neither apple nor pine in pineapple.

• Verb+particle: When a house burns up, it burns down. You fill in a form by filling it out and an alarm clock goes off by going on.

• Predicate+argument: When the stars are out, they are visible, but when the lights are out, they are invisible. And why, when I wind up my watch, I start it, but when I wind up this essay, I end it?

Q4: Can machines be as good as humans in translation quality?

• We know there are things that even humans cannot translate “perfectly”.

• For things that humans can translate, will machines be ever as good as humans in translation quality?– Never say “never”.– Not in the near future.

Q5: what is MT good for?

• Rough translation: web data• Computer-aided human translation• Translation for limited domain• Cross-lingual IR

• Machine is better than human in: – Speed: much faster than humans– Memory: can easily memorize millions of word/phrase

translations.– Manpower: machines are much cheaper than humans– Fast learner: it takes minutes or hours to build a new system.

Erasable memory – Never complain, never get tired, …

Q6: what’s the MT history? (Based on work by John Hutchins)

• Before the computer: In the mid 1930s, a French-Armenian Georges Artsrouni and a Russian Petr Troyanskii applied for patents for ‘translating machines’.

• The pioneers (1947-1954): the first public MT demo was given in 1954 (by IBM and Georgetown University).

• The decade of optimism (1954-1966): ALPAC (Automatic Language Processing Advisory Committee) report in 1966: "there is no immediate or predictable prospect of useful machine translation."

A brief history of MT (cont)

• The aftermath of the ALPAC report (1966-1980): a virtual end to MT research

• The 1980s: Interlingua, example-based MT

• The 1990s: Statistical MT

• The 2000s: Hybrid MT

Q7: where are we now?

• Huge potential/need due to the internet, globalization and international politics.

• Quick development time due to SMT, the availability of parallel data and computers.

• Translation is reasonable for language pairs with a large amount of resource.

• Start to include more “minor” languages.

MT in a nutshell

• What is the ultimate goal of translation?• What are the perfect translations? • Can human always get perfect

translations? • Can machines be as good as humans?• What are MT good for?• What is the MT history?• Where are we now?

Outline

• MT in a nutshell

• Major challenges in MT

• Major approaches

• Evaluation of MT systems

Major challenges in MT

Major challenges

• Getting the right words:– Choosing the correct root form– Getting the correct inflected form– Inserting “spontaneous” words

• Putting the words in the correct order:– Word order: SVO vs. SOV, …– Unique constructions: – Divergence

Lexical choice

• Homonymy/Polysemy: bank, run

• Concept gap: no corresponding concepts in another language: go Greek, go Dutch, fen sui, lame duck, Chinese idioms, …

• Coding (Concept lexeme mapping) differences:– More distinction in one language: e.g., kinship

vocabulary.– Different division of conceptual space:

More distinction: the “cousin” example

Translations: male/female, older or younger than the speaker, from mother or father’s side, parent’s brother or sister’s child:

• 堂兄 (tang xiong): father’s brother’s son, who is older• 堂弟 (tang di): father’s brother’s son, who is younger• 表兄 (biao xiong):

– 姨表兄 (yi biao xiong): mother’s sister’s son, who is older– 舅表兄 (jiu biao xiong): mother’s brother’s son, who is older– 姑表兄 (gu biao xiong): father’s sister’s son, who is older

• 表弟 (biao di): mother’s sibling’s son, who is younger– …

• …16+ translations 8 4.

More distinction: the “aunt” example

Mother’s or father’s side, is a sister or is a brother’s wife:

• 姑妈： father’s sister

• 伯母： father’s younger brother’s wife

• 婶： father’s elder brother’s wife

• 姨妈： mother’s sister

• 舅妈： mother’s brother’s wife

A large happy family

• 三姨姥爷： mother’s mother’s 3rd sister’s husband.

• 3rd among all the sisters, or among all the siblings, or among all the members in the extended family with the same “bei fen”.

• Same “bei fen” ： same distance to the root node of a family tree.

Sapir-Whorf hypothesis

• Edward Sapir (1884-1936): American linguist and anthropologist.

• Benjamin Lee Whorf (1897-1941): Sapir’s student.

• Linguistic determinism/relativism: the language we use to some extent determines the way in which we view and think about the world around us.

• Strong determinism: language determines thought, that language and thought are identical.

• Weak determinism: thought is merely affected by or influenced by our language.

Sapir-Whorf hypothesis (cont)

• “Snow”– Whorf: the number of words for “snow” the Inuit

people have for ‘snow‘ Inuit people treat snow differently than someone who lives in a less snow-dependent environment.

– Pullum (1991): Other languages transmit the same ideas using phrases instead of single words.

• My personal experience: – Preference of one language over the other:

Color coding study

• Hypothesis: if one language categorizes color differently than another language, then the different groups should perceive it differently also.

• A 1970 study– Task: give people a sample of 160 colors and ask them to sort it.– People: English speakers (blue-green distinction) and Berinmo

speaker (nol-wor distinction)– the Berinmo speakers were better at matching colors across

their nol, wor categories than across the English blue and green categories and English speakers were better at matching colors across blue and green than across the Berinmo nol and wor (Sawyer, 1999).

Sapir-Whorf hypothesis (cont)

• What’s the relation between language, thought, and cultural perception of reality?

• Does language affect thought? If so, to what degree?

• The hypothesis is still under much debate.

Major challenges

• Getting the right words:– Choosing the correct root form– Getting the correct inflected form– Inserting “spontaneous” words

• Putting the words in the correct order:– Word order: SVO vs. SOV, …– Unique construction: – Structural divergence

Choosing the appropriate inflection

• Inflection: gender, number, case, tense, …

• Ex:– Number: Ch-Eng: all the concrete nouns: ch_book book, books– Gender: Eng-Fr: all the adjectives– Case: Eng-Korean: all the arguments– Tense: Ch-Eng: all the verbs: ch_buy buy, bought, will buy

Inserting spontaneous words• Function words:

– Determiners: Ch-Eng: ch_book a book, the book, the books, books

– Prepositions: Ch-Eng: … ch_November … in November

– Relative pronouns: Ch-Eng: … ch_buy ch_book de ch_person the person who bought /book/

– Possessive pronouns: Ch-Eng: ch_he ch_raise ch_hand He raised his hand(s)

– Conjunction: Eng-Ch: Although S1, S2 ch_although S1, ch_but S2

– …

Inserting spontaneous words (cont)

• Content words:– Dropped argument: Ch-Eng: ch_buy le ma Has Subj bought Obj?

– Chinese First name: Eng-Ch: Jiang … ch_Jiang ch_Zemin …

– Abbreviation, Acronyms: Ch-Eng: ch_12 ch_big the 12th National Congress of the

CPC (Communist Party of China)

– …

Major challenges

• Getting the right words:– Choosing the correct root form– Getting the correct inflected form– Inserting “spontaneous” words

• Putting the words in the correct order:– Word order: SVO vs. SOV, …– Unique construction: – Structural divergence

Word order

• SVO, SOV, VSO, …• VP + PP PP VP• VP + AdvP AdvP + VP

• Adj + N N + Adj• NP + PP PP NP• NP + S S NP

• P + NP NP + P

“Unique” Constructions

• Overt wh-movement: Eng-Ch:– Eng: Why do you think that he came yesterday?– Ch: you why think he yesterday come ASP?– Ch: you think he yesterday why come?

• Ba-construction: Ch-Eng– She ba homework finish ASP She finished her

homework.– He ba wall dig ASP CL hole He digged a hole in

the wall.– She ba orange peel ASP skin She peeled the

orange’s skin.

Translation divergences(based on Bonnie Dorr’s work)

• Thematic divergence: I like Mary S: Marta me gusta a mi (‘Mary pleases me’)

• Promotional divergence: John usually goes home S: Juan suele ira casa (‘John tends to go home’)

• Demotional divergence: I like eating G: Ich esse gern (“I eat likingly)

• Structural divergence: John entered the house S: Juan entro en la casa (‘John entered in the house’)

Translation divergences (cont)

• Conflational divergence: I stabbed John S: Yo le di punaladas a Juan (‘I gave knife-

wounds to John’)

• Categorial divergence: I am hungry G: Ich habe Hunger (‘I have hunger’)

• Lexical divergence: John broke into the room S: Juan forzo la entrada al cuarto (‘John forced

the entry to the room’)

Outline

• MT in a nutshell

• Major challenges in MT

• Major approaches

• Evaluation of MT systems

How humans do translation?

• Learn a foreign language:– Memorize word translations– Learn some patterns: – Exercise:

• Passive activity: read, listen• Active activity: write, speak

• Translation:– Understand the sentence– Clarify or ask for help (optional)– Translate the sentence

Training stage

Decoding stage

Translation lexicon

Templates, transfer rules

Parsing, semantics analysis?

Interactive MT?

Word-level? Phrase-level? Generate from meaning?

Reinforced learning?Reranking?

What kinds of resources are available to MT?

• Translation lexicon: – Bilingual dictionary

• Templates, transfer rules:– Grammar books

• Parallel data, comparable data• Thesaurus, WordNet, FrameNet, …

• NLP tools: tokenizer, morph analyzer, parser, …

More resources for major languages, less for “minor” languages.

Major approaches

• Transfer-based

• Interlingua

• Example-based (EBMT)

• Statistical MT (SMT)

• Hybrid approach

The MT triangle

word Word

Meaning

Transfer-based

Phrase-based SMT, EBMT

Word-based SMT, EBMT

(interlingua)

Ana

lysi

sS

ynthesis

Transfer-based MT

• Analysis, transfer, generation:1. Parse the source sentence2. Transform the parse tree with transfer rules3. Translate source words4. Get the target sentence from the tree

• Resources required:– Source parser– A translation lexicon– A set of transfer rules

• An example: Mary bought a book yesterday.

Transfer-based MT (cont)

• Parsing: linguistically motivated grammar or formal grammar?

• Transfer: context-free rules? Additional constraints on the rules? Apply at most one rule at each level? How are rules created?

• Translating words: word-to-word translation?• Generation: using LM or other additional

knowledge?

• How to create the needed resources automatically?

Interlingua

• For n languages, we need n(n-1) MT systems.• Interlingua uses a language-independent

representation. • Conceptually, Interlingua is elegant: we only

need n analyzers, and n generators.

• Resource needed:– A language-independent representation– Sophisticated analyzers– Sophisticated generators

Interlingua (cont)

• Questions:– Does language-independent meaning representation

really exist? If so, what does it look like?– It requires deep analysis: how to get such an

analyzer: e.g., semantic analysis– It requires non-trivial generation: How is that done?– It forces disambiguation at various levels: lexical,

syntactic, semantic, discourse levels.– It cannot take advantage of similarities between a

particular language pair.

Example-based MT

• Basic idea: translate a sentence by using the closest match in parallel data.

• First proposed by Nagao (1981).• Ex:

– Training data:• w1 w2 w3 w4 w1’ w2’ w3’ w4’• w5 w6 w7 w5’ w6’ w7’• w8 w9 w8’ w9’

– Test sent:• w1 w2 w6 w7 w9 w1’ w2’ w6’ w7’ w9’

EMBT (cont)

• Types of EBMT:– Lexical (shallow)– Morphological / POS analysis – Parse-tree based (deep)

• Types of data required by EBMT systems:– Parallel text– Bilingual dictionary – Thesaurus for computing semantic similarity– Syntactic parser, dependency parser, etc.

EBMT (cont)

• Word alignment: using dictionary and heuristics exact match

• Generalization: – Clusters: dates, numbers, colors, shapes, etc. – Clusters can be built by hand or learned automatically.

• Ex:– Exact match: 12 players met in Paris last Tuesday

12 Spieler trafen sich letzen Dienstag in Paris

– Templates: $num players met in $city $time $num Spieler trafen sich $time in $city

Statistical MT

• Basic idea: learn all the parameters from parallel data.

• Major types:– Word-based– Phrase-based

• Strengths:– Easy to build, and it requires no human knowledge– Good performance when a large amount of training data is

available.

• Weaknesses:– How to express linguistic generalization?

Comparison of resource requirement

Transfer-based

Interlingua EBMT SMT

dictionary + + +

Transfer rules

+

parser + + + (?)

semantic

analyzer

+

parallel data + +

others Universal representation

thesaurus

Hybrid MT• Basic idea: combine strengths of different approaches:

– Syntax-based: generalization at syntactic level– Interlingua: conceptually elegant– EBMT: memorizing translation of n-grams; generalization at various level.– SMT: fully automatic; using LM; optimizing some objective functions.

• Types of hybrid HT:– Borrowing concepts/methods:

• SMT from EBMT: phrase-based SMT; Alignment templates• EBMT from SMT: automatically learned translation lexicon• Transfer-based from SMT: automatically learned translation lexicon, transfer rules;

using LM• …

– Using two MTs in a pipeline:• Using transfer-based MT as a preprocessor of SMT

– Using multiple MTs in parallel, then adding a re-ranker.

Outline

• MT in a nutshell

• Major challenges in MT

• Major approaches

• Evaluation of MT systems

Evaluation

• Unlike many NLP tasks (e.g., tagging, chunking, parsing, IE, pronoun resolution), there is no single gold standard for MT.

• Human evaluation: accuracy, fluency, …– Problem: expensive, slow, subjective, non-reusable.

• Automatic measures:– Edit distance– Word error rate (WER), Position-independent WER (PER)– Simple string accuracy (SSA), Generation string accuracy (GSA)– BLEU

Edit distance

• The Edit distance (a.k.a. Levenshtein distance) is defined as the minimal cost of transforming str1 into str2, using three operations (substitution, insertion, deletion).

• Let the operation cost be subCost, insCost, and delCost, respectively.

• Let |Str1|=m and |Str2|=n, D(i,j) stores the edit distance of converting str1[1..i] to str2[1..j].

• D(m,n) is the answer that we are looking for.

• Use DP and the complexity is O(m*n).

Calculating edit distance

• D(0, 0) = 0• D(i, 0) = delCost * i• D(0, j) = insCost * j

• D(i+1, j+1) = min( D(i,j) + sub, D(i+1, j) + insCost, D(i, j+1) + delCost)

sub = 0 if str1[i+1]=str2[j+1] = subCost otherwise

An example

• Sys: w1 w2 w3 w4• Ref: w1 w3 w2• All three costs are 1.

• Edit distance=2

0 1 2 3

1 0 1 2

2 1 1 1

3 2 1 2

4 3 2 2

w1 w3 w2

w1

w2

w3

w4

WER, PER, and SSA

• WER (word error rate) is edit distance, divided by |Ref|.• PER (position-independent WER): same as WER but

disregards word ordering• SSA (Simple string accuracy) = 1 - WER

• Previous example:– Sys: w1 w2 w3 w4– Ref: w1 w3 w2– Edit distance = 2– WER=2/3– PER=1/3– SSA=1/3

Generation string accuracy (GSA)

Example:Ref: w1 w2 w3 w4

Sys: w2 w3 w4 w1

Del=1, Ins=1 SSA=1/2Move=1, Del=0, Ins=0 GSA=3/4

|Re|1

f

SubDelInsMoveGSA

BLEU

• Proposal by Papineni et. al. (2002)• Most widely used in MT community.• BLEU is a weighted average of n-gram precision

(pn) between system output and all references, multiplied by a brevity penalty (BP).

)1

(...***

*

21

1

NwwhenpppBP

pBPBLEU

nN

N

N

n

wn

n

N-gram precision

• N-gram precision: the percent of n-grams in the system output that are correct.

• Clipping: – Sys: the the the the the the– Ref: the cat sat on the mat– Unigram precision: – Max_Ref_count: the max number of times a

ngram occurs in any single reference translation.

)_Re_,min( CountfMaxcountCountclip

N-gram precision

i.e. the percent of n-grams in the system output that are correct (after clipping).

SysS Sngram

SysS Sngramclip

n ngramCount

ngramCount

p)(

)(

Brevity Penalty

• For each sent si in system output, find closest matching reference ri (in terms of length).

• Longer system output is already penalized by the n-gram precision measure.

otherwisee

rcifBP cr /1

1

|||,| i

ii

i rrscLet

An example

• Sys: The cat was on the mat• Ref1: The cat sat on a mat• Ref2: There was a cat on the mat

• Assuming N=3

• p1=5/6, p2=3/5, p3=1/4, BP=1 BLEU=0.50

• What if N=4?

Summary

• MT in a nutshell

• Major challenges in MT– Choose the right words (root form, inflection,

spontaneous words)– Put them in right positions (word order, unique

constructions, divergences)

Summary (cont)

• Major approaches– Transfer-based MT– Interlingua– Example-based MT– Statistical MT– Hybrid MT

• Evaluation of MT systems– Edit distance– WER, PER, SSA, GSA– BLEU