Joke Daems PhD student Lieve Macken, Sonia Vandepitte, Robert Hartsuiker Comparing HT and PE using advanced research tools.

Joke DaemsPhD student

Lieve Macken, Sonia Vandepitte, Robert Hartsuiker

Comparing HT and PE using advanced research tools

Overview

• Experimental setup• Process

– Speed– Translation units– Cognitive load– Gaze behavior– External resources

• Product– Acceptability– Adequacy – Influence of MT on PE quality– Overall quality: Influence of external resources

Overview

• Experimental setup• Process

– Speed– Translation units– Cognitive load– Gaze behavior– External resources

• Product– Acceptability– Adequacy – Influence of MT on PE quality– Overall quality: Influence of external resources

Experimental setup

• Text selection• Participants• Session• Tools• Data collected

Text selection

• 15 newspaper articles (+/- 150 words) from Newsela.com

• Readability metrics• Translation problems• MT quality 8 texts selected

Participants

• 10 master’s students of translation (pass general translation exam)

• No previous experience post-editing• Reward: 2 coupons of 50 euros each

Session overview• Session 1:– Survey – LexTALE test (proficiency)– Warmup task– 2 texts HT, 2 texts PE

• Session 2:– Warmup task– 2 texts PE, 2 texts HT– Retrospection– Survey

Experimental setup: DesignParticipant P1 P3 P5 P7 P9 P2 P4 P6 P8 P10

Session1 task1 PE_1 PE_8 PE_7 PE_6 PE_5 HT_1 HT_8 HT_7 HT_6 HT_5

task2 PE_2 PE_1 PE_8 PE_7 PE_6 HT_2 HT_1 HT_8 HT_7 HT_6

task3 HT_3 HT_2 HT_1 HT_8 HT_7 PE_3 PE_2 PE_1 PE_8 PE_7

task4 HT_4 HT_3 HT_2 HT_1 HT_8 PE_4 PE_3 PE_2 PE_1 PE_8

Session2 task5 HT_5 HT_4 HT_3 HT_2 HT_1 PE_5 PE_4 PE_3 PE_2 PE_1

task6 HT_6 HT_5 HT_4 HT_3 HT_2 PE_6 PE_5 PE_4 PE_3 PE_2

task7 PE_7 PE_6 PE_5 PE_4 PE_3 HT_7 HT_6 HT_5 HT_4 HT_3

task8 PE_8 PE_7 PE_6 PE_5 PE_4 HT_8 HT_7 HT_6 HT_5 HT_4

Tools

• EyeLink (eye-tracking)• Casmacat (keystrokes + compatibility EyeLink)• Inputlog (keystrokes + logging of external

resources)

Data

• 80 sessions– 40 HT, 40 PE– 10 for each text– 8 for each translator

• Keystroke logging (Casmacat + Inputlog) & eye-tracking

• Pre- & post surveys• Annotation of problems

Overview

• Experimental setup• Process

– Speed– Translation units– Cognitive load– Gaze behavior– External resources

• Product– Acceptability– Adequacy – Influence of MT on PE quality– Overall quality: Influence of external resources

Translation speed

523.43 ms (± 202.14 standard error) more in HT

Production units

0,1 PU (± 0,019 se) more per ST token in HT 1295ms (± 227 se) more per PU in HT

Cognitive load

• Post-task survey: Which translation method was most tiring?–5 respondents: HT and PE equally tiring–1 respondent: PE most tiring–4 respondents: HT most tiring

Cognitive load

• What makes HT so tiring?– “Having to start from nothing”– “Insecurity, I need to double-check everything because

there’s no basic structure as with PE”– “Looking up synonyms, finding the correct words”

• What makes PE so tiring?– “Letting go of the MT output”– “Noticing errors in the MT output”– “Making sure the structure is still okay after fixing one

part of the sentence”

Gaze behavior: duration

Longer fixations on TT (cognitive load?)Difference greatest for HT (harder to understand MT

without ST? No effect found for MT quality)

Gaze behaviour: # fixations

More fixations on TT Difference greatest for PE (MT output? No effect found for MT

quality)

Gaze behaviour: # fixations ST and TT

ST: 2,3 (± 0,4 se) more fixations per ST token in HTTT: 1,3 (± 0,5 se) fewer fixations per ST token in HT

External resources

External resources

297.46 ms (± 105.36 standard error) more time in HTNo significant effect of MT quality

Overview

• Experimental setup• Process

– Speed– Translation units– Cognitive load– Gaze behavior– External resources

• Product– Acceptability– Adequacy – Influence of MT on PE quality– Overall quality: Influence of external resources

Quality: acceptability• Task in itself no significant predictor• Participant * Task: R²=0,44, p<0,001

Quality: adequacy

• Task in itself no significant predictor• Participant * Task no significant predictor• Post-editing adequacy error score: negatively

influenced by MT acceptability error score

PE adequacy ~ MT acceptability

Having to solve many acceptability issues might make students more aware of adequacy issues as well.

Overall quality: influence of external resources

Strategies used when translating not always successful when post-editing

Conclusions• PE faster than HT• More & longer production units in HT• HT perceived as more tiring• Longer fixations on TT (more so in HT)• More fixations on TT (more so in PE)• HT: more fixations ST, PE: more fixations TT• More time in external resources in HT• Acceptability more dependent on participant than task• PE adequacy influenced by MT acceptability• Consulting external resources more effective in HT

In sum

PE is faster than HT, cognitively less demanding, and requires less usage of external resources to obtain a product of comparable quality.

But, students need to be trained to better use external resources when post-editing.

Future research

• More fine-grained analysis:– Segment level– Comparison per participant– Comparison per text– Problem-solving strategies– Difficult passages

• Repeat with professional translators

Thank you for your time

Questions?Suggestions?

[email protected]