META-NET is partially supported by the DG INFSO of the European Commission through the NoE project T4ME of the Seventh Framework Programme. Grant Agreement.

META-NET is partially supported by the DG INFSO of the European Commission through the META-NET is partially supported by the DG INFSO of the European Commission through the NoE project T4ME of the Seventh Framework Programme. Grant Agreement No.: 249119NoE project T4ME of the Seventh Framework Programme. Grant Agreement No.: 249119

Dimitris Mavroeidis

Institute for Language and Speech Processing

Greece

Divide and Translate: Using Document Classification to improve

Statistical Machine Translation

DateDivide and TranslateTitle of

Presentation 2

Language Models

Counts n-grams

Models a coprus

Compare with documents outside the corpus

Calculate perplexity to assess similarity

2


Presentation 3

EUROVOC

A multi-lingual thesaurus (eurovoc.europa.eu)

Created by the European Commission

Used primarily for indexing purposes

21 top-level domains:

04 POLITICS

08 INTERNATIONAL RELATIONS

…

GEOGRAPHY

INTERNATIONAL ORGANIZATIONS

Date 3


Presentation 4

Classification using Language Models

Existing corpora with domain labels

Train one LM per domain

Calculate perplexity of incoming document against all LMs

The lowest perplexity is an indication that the text belongs to the corresponding domain

Let’s see if that’s true…


Presentation 5

Context in Machine Translation: Research

Corpus: JRC-Acquis (legal documents of the EU)

Labeled with one or more EUROVOC terms

Each term belongs to one of the 21 domains

Each document can be classified to one of these domains

Date 5


Presentation 6

Corpus

Date 6

Language ISO code

No of texts

Text body Signatures AnnexesTotal No words (text +

signatures + annexes):

Total No words

Total No characters

Average No words

Total No words

Total No words

bg 11384 16140819 1045226711417.85 2170075 14114612 32425506

cs 21438 22843279 1489729811065.55 7225300 16763733 46832312

da 23624 31459627 2134681351331.68 2629786 16855213 50944626

de 23541 32059892 2327486751361.87 2542149 16327611 50929652

el 23184 36453749 2395835431572.37 2973574 16459680 55887003

en 23545 34588383 2106920591469.03 3198766 17750761 55537910

es 23573 38926161 2380167561651.3 3490204 19716243 62132608

et 23541 24621625 1927007041045.9 1336051 14995748 40953424

fi 23284 24883012 2121789641068.67 2677798 12547171 40107981

fr 23627 39100499 2347582901654.91 3021013 19978920 62100432

hu 22801 28602380 2138046141254.44 2529488 15056496 46188364

it 23472 35764670 2306770131523.72 3120797 18331535 57217002

lt 23379 26937773 1994382581152.22 2436585 15018484 44392842

lv 22906 27592514 1964520511204.6 1673124 15437969 44703607

mt 10545 20926909 1289067481984.53 1336042 15620611 37883562

nl 23564 35265161 2319635391496.57 3039580 18467115 56771856

pl 23478 29713003 2144640261265.57 2513141 17027393 49253537

pt 23505 37221668 2274994181583.56 3034308 19350227 59606203

ro 6573 9186947 605373011397.68 514296 11185842 20887085

ro 19211 30832212 1826312771604.92 --- --- 30832212

sk 21943 26792637 1799204341221.01 3227852 16190546 46211035

sl 20642 27702305 1786517671342.04 3103193 16837717 47643215

sv 20243 29433037 1990044011453.99 2575771 14965384 46974192

Total 463792 636216050 42889623481387.23 60368893 358999011 1055583954


Presentation 7

Partitioning JRC-Acquis

For each document

Create 21 counters For each EUROVOC term in the document

Find top-level domains

Add 1 to the corresponding counter Find the domain with the most counts

Find the domain with the most counts

Classify document

Date 7


Presentation 8

Partitioning JRC-Acquis

Date 8

JRC-Acquis Partitioning

04 POLITICS

08 INTL. RELATIONS

76 INTL. ORGANIZATIONS

72 GEOGRAPHY

.

.

.

Sub-corporaCorpus


Presentation 9

Classes info

Date 9

Domain Description Lines Words04 POLITICS 25.197 722.63508 INTERNATIONAL RELATIONS 69.057 1.832.10010 EUROPEAN COMMUNITIES 164.223 4.310.70212 LAW 31.885 917.51016 ECONOMICS 35.207 1.114.05220 TRADE 100.306 2.485.81424 FINANCE 86.440 2.015.94328 SOCIAL QUESTIONS 47.343 1.387.419

32 EDUCATION AND COMMUNICATIONS 22.313 701.76136 SCIENCE 23 20340 BUSINESS AND COMPETITION 51.701 1.388.13344 EMPLOYMENT AND WORKING CONDITIONS 17.644 473.97948 TRANSPORT 26.696 733.19152 ENVIRONMENT 39.448 1.141.31356 AGRICULTURE, FORESTRY AND FISHERIES 88.165 2.235.48060 AGRI-FOODSTUFFS 65.331 1.449.96964 PRODUCTION, TECHNOLOGY AND RESEARCH 50.641 1.313.69566 ENERGY 7.604 226.48668 INDUSTRY 32.874 1.115.37972 GEOGRAPHY 88.119 2.348.37376 INTERNATIONAL ORGANISATIONS 9.666 271.810


Presentation 10

Experimentation

Trained Domain-specific and Generic Language Models

SRILM

N-gram order: 3

Smoothing: Knesser-Ney

Trained Domain and Generic Translation Models

MOSES

Maximum sentence length: 120

Tuning (MERT) on each domain

Evaluate whether the selected domain corresponds to the best BLEU score for the respective translation

Date 10


Presentation 11

Creating individual domains

11


Presentation 12

Creating generic domain

12

+ +04

trainingset

08 training

set

72 training

set

76 training

set. . .

~~~~~~~~~~~~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~~~~~

GENERICTM

GENERICLM

GENERIC test+dev set

GENERIC CORPUS

GENERIC training set


Presentation 13

Results

13

Domain 04-Politics

DOMAIN PERPLEXITYBLEU

SCORE04-Politics 64.552 0.5416

10-European Com. 80.999 0.341708-Int. Relations 99.208 0.3383

28-Social Q. 102.714 0.349152-Environment 107.159 0.3502

12-Law 107.175 0.358020-Trade 109.092 0.3206

32-Education 117.673 0.360764-Pr.Tech.&Res. 117.897 0.3282

24-Finance 120.497 0.323356-Agriculture 122.930 0.3120

40-Business 123.195 0.321344-Employment 123.824 0.3423

16-Economics 126.787 0.339572-Geography 127.058 0.3186

48-Transport 129.126 0.326976-Intl. Organ. 140.126 0.3374

60-Agri-Foodstuffs 149.198 0.298568-Industry 152.372 0.3216

ALL 0.3574

Domain 08-International Relations


SCORE08-Int. Relations 47.884 0.5777

10-European Com. 77.843 0.368316-Economics 112.017 0.3706

28-Social Q. 114.596 0.364524-Finance 115.043 0.3425

56-Agriculture 116.436 0.339272-Geography 116.737 0.3435

20-Trade 117.706 0.334404-Politics 118.231 0.3801

32-Education 119.815 0.378952-Environment 123.425 0.3687

12-Law 124.239 0.366140-Business 127.654 0.3411

64-Pr.Tech.&Res. 127.786 0.340160-Agri-Foodstuffs 129.150 0.3171

44-Employment 132.615 0.350148-Transport 135.504 0.3349

76-Intl. Organ. 139.757 0.357868-Industry 147.744 0.3384

ALL 0.3876


Presentation 14

Results (2)

14

Domain 10-European Communities


SCORE10-European Com. 47.636 0.5766

24-Finance 104.841 0.349308-Int. Relations 107.798 0.3560

28-Social Q. 118.812 0.365104-Politics 125.323 0.3813

44-Employment 126.737 0.362832-Education 126.994 0.3868

12-Law 127.538 0.371652-Environment 128.789 0.3663

40-Business 129.283 0.346616-Economics 129.351 0.369272-Geography 132.749 0.3433

64-Pr.Tech.&Res. 132.963 0.341520-Trade 135.006 0.3228

48-Transport 140.200 0.341656-Agriculture 141.078 0.328376-Intl. Organ. 147.871 0.3629


ALL 0.3906

Domain 12-Law


SCORE12-Law 48.386 0.5654

10-European Com. 72.444 0.371528-Social Q. 95.177 0.3792

04-Politics 104.735 0.387908-Int. Relations 107.970 0.355944-Employment 108.837 0.3791

32-Education 110.842 0.387552-Environment 110.846 0.3686

24-Finance 116.578 0.335020-Trade 119.054 0.3280

40-Business 121.122 0.341772-Geography 121.904 0.3394

64-Pr.Tech.&Res. 123.383 0.343348-Transport 127.228 0.3431

16-Economics 127.581 0.360256-Agriculture 131.967 0.329876-Intl. Organ. 140.408 0.3616


ALL 0.3772


Presentation 15

Results (3)

15

Domain 16-Economics


SCORE16-Economics 75.695 0.5961

10-European Com. 129.686 0.438872-Geography 135.265 0.4461

40-Business 144.845 0.430908-Int. Relations 148.029 0.4339

24-Finance 156.598 0.424932-Education 157.289 0.4505

28-Social Q. 158.574 0.438352-Environment 162.246 0.433344-Employment 165.868 0.4383

12-Law 170.435 0.442104-Politics 173.194 0.4403

48-Transport 173.647 0.408456-Agriculture 183.242 0.407876-Intl. Organ. 187.693 0.4303

68-Industry 192.026 0.434964-Pr.Tech.&Res. 198.253 0.3772

20-Trade 214.334 0.397860-Agri-Foodstuffs 222.229 0.3898

ALL 0.4436

Domain 20-Trade


SCORE20-Trade 20.321 0.6993

56-Agriculture 45.361 0.485960-Agri-Foodstuffs 45.952 0.4812

72-Geography 47.882 0.480310-European Com. 62.000 0.4446

08-Int. Relations 63.673 0.448228-Social Q. 64.489 0.465568-Industry 65.617 0.4709

32-Education 69.479 0.491564-Pr.Tech.&Res. 69.571 0.434952-Environment 70.296 0.4713

24-Finance 72.645 0.437748-Transport 73.595 0.4518

40-Business 74.429 0.447712-Law 74.671 0.4597

04-Politics 76.775 0.485476-Intl. Organ. 77.886 0.456516-Economics 80.977 0.4516

44-Employment 81.858 0.4387ALL 0.5039


Presentation 16

Results (4)

16

Domain 24-Finance


SCORE24-Finance 36.668 0.6569

10-European Com. 74.678 0.442108-Int. Relations 118.026 0.4142

16-Economics 142.699 0.437440-Business 147.090 0.4054

72-Geography 148.029 0.411232-Education 149.387 0.4372

44-Employment 163.200 0.414264-Pr.Tech.&Res. 164.124 0.4019

20-Trade 165.852 0.400112-Law 166.816 0.4201

04-Politics 168.382 0.436656-Agriculture 170.829 0.398976-Intl. Organ. 172.822 0.4213

28-Social Q. 172.843 0.411252-Environment 173.045 0.4210

48-Transport 174.151 0.396360-Agri-Foodstuffs 178.271 0.3885

68-Industry 192.723 0.4046ALL 0.4616

Domain 28-Docial Questions


SCORE28-Social Q. 45.506 0.5800

10-European Com. 83.198 0.361556-Agriculture 88.334 0.3658

52-Environment 94.853 0.380212-Law 101.534 0.3766

20-Trade 102.380 0.334864-Pr.Tech.&Res. 105.292 0.3493

32-Education 107.702 0.376304-Politics 108.877 0.3731

72-Geography 110.176 0.340808-Int. Relations 110.400 0.343944-Employment 110.673 0.3570

48-Transport 110.821 0.349616-Economics 118.792 0.3607

60-Agri-Foodstuffs 119.393 0.323840-Business 121.580 0.333124-Finance 123.992 0.3329

76-Intl. Organ. 126.679 0.357868-Industry 133.215 0.3429

ALL 0.3829


Presentation 17

Results (5)

17

Domain 32-Education


SCORE32-Education 60.401 0.5981


08-Int. Relations 115.880 0.376712-Law 116.422 0.4022

40-Business 118.794 0.372720-Trade 122.841 0.3516

24-Finance 123.094 0.368852-Environment 124.125 0.3901

16-Economics 125.142 0.395804-Politics 126.635 0.3944

48-Transport 126.920 0.360944-Employment 128.339 0.3856

64-Pr.Tech.&Res. 128.441 0.356072-Geography 129.539 0.366056-Agriculture 130.643 0.360776-Intl. Organ. 140.728 0.3821

68-Industry 142.599 0.380260-Agri-Foodstuffs 159.374 0.3396

ALL 0.4078

Domain 36-Science


SCORE52-Environment 33.351 0.5087

28-Social Q. 37.418 0.498256-Agriculture 39.408 0.4730

10-European Com. 46.291 0.452120-Trade 50.337 0.4257

64-Pr.Tech.&Res. 53.361 0.461232-Education 56.537 0.488348-Transport 57.882 0.4533

44-Employment 59.183 0.459404-Politics 63.028 0.4702

12-Law 63.038 0.462560-Agri-Foodstuffs 65.544 0.4162

08-Int. Relations 66.678 0.428468-Industry 67.845 0.4405


24-Finance 70.578 0.424440-Business 72.025 0.4204

76-Intl. Organ. 75.036 0.4439ALL 0.4748


Presentation 18

Results (6)

18

Domain 40-Business


SCORE40-Business 40.779 0.651368-Industry 85.671 0.4733

72-Geography 91.311 0.453932-Education 93.351 0.4779

10-European Com. 94.110 0.438948-Transport 96.484 0.4385

24-Finance 99.133 0.417308-Int. Relations 100.393 0.4303

20-Trade 106.633 0.417756-Agriculture 113.221 0.4397

28-Social Q. 115.252 0.443716-Economics 121.243 0.4446

52-Environment 127.520 0.443312-Law 130.359 0.4406

60-Agri-Foodstuffs 132.788 0.405644-Employment 134.300 0.4316

04-Politics 138.132 0.450276-Intl. Organ. 140.920 0.4453

64-Pr.Tech.&Res. 144.033 0.4191ALL 0.4702

Domain 44-Employment


SCORE44-Employment 54.777 0.5926


12-Law 106.553 0.396208-Int. Relations 122.386 0.3555

16-Economics 122.830 0.374824-Finance 123.379 0.3435

32-Education 125.096 0.384204-Politics 126.009 0.3769

52-Environment 128.833 0.358340-Business 130.535 0.3448

20-Trade 132.361 0.325164-Pr.Tech.&Res. 133.646 0.3439

48-Transport 135.727 0.346972-Geography 137.461 0.337156-Agriculture 149.723 0.323576-Intl. Organ. 154.331 0.3605


ALL 0.3828


Presentation 19

Results (7)

19

Domain 48-Transport


SCORE48-Transport 40.096 0.6298

10-European Com. 86.438 0.401652-Environment 98.284 0.4192

72-Geography 101.568 0.395964-Pr.Tech.&Res. 101.950 0.3867

56-Agriculture 104.398 0.388420-Trade 105.452 0.3877

32-Education 106.483 0.427340-Business 106.639 0.393228-Social Q. 108.948 0.408268-Industry 109.650 0.4048

08-Int. Relations 114.109 0.391812-Law 116.492 0.4119

04-Politics 119.369 0.412944-Employment 121.457 0.3934

24-Finance 123.002 0.375460-Agri-Foodstuffs 124.095 0.3588

16-Economics 126.691 0.402876-Intl. Organ. 134.055 0.3971

ALL 0.4146

Domain 52-Environment


SCORE52-Environment 40.555 0.5984

56-Agriculture 77.655 0.392628-Social Q. 79.210 0.4077

10-European Com. 80.605 0.381764-Pr.Tech.&Res. 89.542 0.3792

20-Trade 96.919 0.363248-Transport 98.608 0.3801

04-Politics 99.114 0.410912-Law 101.457 0.3959

32-Education 105.314 0.400808-Int. Relations 106.848 0.367444-Employment 110.023 0.3775

72-Geography 112.322 0.359360-Agri-Foodstuffs 112.424 0.3503

24-Finance 113.887 0.357940-Business 114.297 0.3581

16-Economics 116.147 0.384976-Intl. Organ. 119.161 0.3795

68-Industry 121.089 0.3738ALL 0.4042


Presentation 20

Results (8)

20

Domain 56-Agriculture


SCORE56-Agriculture 21.311 0.6965

20-Trade 44.698 0.463872-Geography 48.244 0.4590

60-Agri-Foodstuffs 49.560 0.462528-Social Q. 50.623 0.4868

52-Environment 52.310 0.490410-European Com. 59.105 0.4466

08-Int. Relations 64.748 0.435832-Education 65.439 0.4849

64-Pr.Tech.&Res. 66.492 0.438948-Transport 66.513 0.4456

68-Industry 67.578 0.459840-Business 69.321 0.4388

16-Economics 72.959 0.452224-Finance 73.614 0.4380

12-Law 73.641 0.459844-Employment 76.696 0.4419

04-Politics 77.201 0.467576-Intl. Organ. 80.833 0.4484

ALL 0.5064

Domain 60-Agri-Foodstuffs


SCORE60-Agri-Foodstuffs 12.468 0.7513

20-Trade 29.111 0.553256-Agriculture 30.073 0.574672-Geography 45.996 0.5234

28-Social Q. 51.405 0.524324-Finance 51.536 0.5133

10-European Com. 52.493 0.511364-Pr.Tech.&Res. 54.639 0.502752-Environment 56.174 0.538708-Int. Relations 56.408 0.4983

32-Education 58.535 0.544016-Economics 58.863 0.5084

68-Industry 59.566 0.507848-Transport 61.018 0.4891

40-Business 62.123 0.501012-Law 63.708 0.4981

44-Employment 63.930 0.495876-Intl. Organ. 64.322 0.5015

04-Politics 69.611 0.5371ALL 0.5743


Presentation 21

Results (9)

21

Domain 64-Production Tech. & Research


SCORE64-Pr.Tech.&Res. 26.989 0.6789

76-Intl. Organ. 89.607 0.454620-Trade 104.523 0.4013


52-Environment 129.476 0.441832-Education 142.329 0.436648-Transport 148.674 0.4065

68-Industry 148.929 0.406156-Agriculture 161.790 0.3989

12-Law 165.351 0.417444-Employment 166.413 0.417608-Int. Relations 168.470 0.4022

24-Finance 170.674 0.386804-Politics 178.878 0.4266

40-Business 180.687 0.396460-Agri-Foodstuffs 180.718 0.3729


ALL 0.4533

Domain 68-Energy


SCORE10-European Com. 102.909 0.3678

52-Environment 103.278 0.391528-Social Q. 120.341 0.3704

08-Int. Relations 121.407 0.359740-Business 124.397 0.361124-Finance 128.022 0.3494

32-Education 128.202 0.389148-Transport 128.228 0.3560

64-Pr.Tech.&Res. 133.942 0.349304-Politics 135.144 0.3786

20-Trade 139.109 0.335416-Economics 140.030 0.3678

12-Law 140.222 0.362856-Agriculture 140.689 0.3396

44-Employment 143.205 0.358572-Geography 143.519 0.348176-Intl. Organ. 147.300 0.3774


ALL 0.3837


Presentation 22

Results (10)

22

Domain 68-Industry


SCORE68-Industry 33.509 0.6569

72-Geography 58.345 0.440940-Business 72.790 0.4251

56-Agriculture 76.111 0.414620-Trade 82.786 0.4018

32-Education 101.920 0.433548-Transport 101.944 0.3987

60-Agri-Foodstuffs 110.246 0.365928-Social Q. 124.250 0.3936

10-European Com. 124.879 0.371408-Intl. Relations 132.218 0.381364-Pr.Tech.&Res. 150.997 0.347152-Environment 152.684 0.3849

76-Intl. Organ. 153.576 0.388224-Finance 153.804 0.3494

16-Economics 156.407 0.389912-Law 157.606 0.3774

04-Politics 164.156 0.390944-Employment 178.374 0.3605

ALL 0.4343

Domain 72-Geography


SCORE72-Geography 34.878 0.6754

40-Business 84.797 0.442356-Agriculture 87.246 0.4358

68-Industry 90.483 0.472510-European Com. 93.257 0.4326

20-Trade 95.614 0.425508-Intl. Relations 109.182 0.4285

16-Economics 110.311 0.448432-Education 110.528 0.4652

28-Social Q. 111.068 0.438448-Transport 115.785 0.4186

60-Agri-Foodstuffs 117.086 0.403152-Environment 121.141 0.4360

24-Finance 121.150 0.408512-Law 129.067 0.4290

76-Intl. Organ. 134.859 0.438544-Employment 137.654 0.4172

04-Politics 142.644 0.439064-Pr.Tech.&Res. 148.283 0.3877

ALL 0.4793


Presentation 23

Results (11)

23

Domain 76-International Organisations


SCORE76-Intl. Organ. 52.157 0.6045

64-Pr.Tech.&Res. 76.672 0.396310-European Com. 115.134 0.3852

20-Trade 125.731 0.352228-Social Q. 127.764 0.3801

52-Environment 140.542 0.385832-Education 141.639 0.3904

8-Intl. Relations 155.090 0.374568-Industry 162.352 0.3628

12-Law 163.230 0.375740-Business 163.792 0.3585

48-Transport 167.686 0.36554-Politics 170.762 0.3785

72-Geography 171.254 0.363224-Finance 173.396 0.3423

16-Economics 178.379 0.369144-Employment 178.388 0.3656

56-Agriculture 180.153 0.346060-Agri-Foodstuffs 203.256 0.3229

ALL 0.4019


Presentation 24

Future work

Better classification algorithms (for new documents)

Application on other corpora (e.g. Europarl, SE-Times)

More language pairs

Add the domain-specific corpus multiple times

Different approaches on partitioning JRC-Acquis

Date 24

META-NET is partially supported by the DG INFSO of the European Commission through the NoE project T4ME of the Seventh Framework Programme. Grant Agreement.

Documents

date divide

translatetitle of presentation

document date

domains date

corpus date

respective translation

generic domain

corresponding domain