Treating translation quality metrics as business intelligence by Samuel Läubli (Autodesk) and Hyunjoo Han (Autodesk); moderated by Clove Lynch and Eduardo D'Antonio (VMWare)

© 2015 Autodesk

Treating translation quality metrics as business intelligenceSamuel Läubli, Hyunjoo Han

© 2015 Autodesk

Translation Lifecycle & Data Collection Translation Quality Criteria Machine Translation Linguistic Quality Dashboard

Flow of this presentation

© 2015 Autodesk

New

Translate

Review

Correction

Deprecate

Translation Lifecycle

© 2015 Autodesk

New• Content Type• Timestamp• % of change

Translate• Content Type• Wordcount• Raw MT usefulness• Level of TM/MT

Quality• Timestamp• Frequency

Review• Content Type• Wordcount• Review Score• Ratings per

criteria

Correction• Content Type• Completeness• Frequency

Deprecate• Content Type• Timestamp

Translation Lifecycle – Data Collection

© 2015 Autodesk

Quality Criteria

Concrete ErrorConcrete Error attributes are objective and countable ratings. The score is based on the number of errors found per category and severity level assigned to the error.

Global Language AttributesGlobal Language attributes are subjective language quality attributes that are measured by a rating scale rather than by counting errors.

Final Quality ScoreThe final quality score is calculated as follows: - Marketing: Concrete x 60% + Global Language x

40%- Software: Concrete x 75% + Global Language x

25%

ConcreteError

Global Language Attributes

• Accuracy• Style• Language• Terminology• Software

Consistency• Locale• Layout (DTP)**

• Fluency• Suitability/

Adequacy• General Writing

Style• Creativity

** marketing contents only

© 2015 Autodesk

In-house MT systems first piloted in 2008 Hypothesis: Professional translators will save time by post-

editing MT rather than translating from scratch Lots of skepticism and pushback from translators, vendors,

and internal stakeholders Formal, comprehensive productivity test in 2009

Main finding: post-editing brings significant productivity gains Findings and data shared with academic community (Plitt & Masselot,

2010). Big interest and many follow-up studies (e.g., Green et al., 2013; Läubli et al., 2013)

Sharing of data was key to convincing our stakeholders

Machine Translation – Background

© 2015 Autodesk

MT on knowledge.autodesk.com

© 2015 Autodesk

Machine Translation – Metrics How good is a machine translation engine? Suitability of metrics depends on objective:

Translator productivity: distance-based metrics ? Comprehensibility of raw MT for end-users: precision/recall-based

metrics ? The latter is a lot harder to measure – and becomes more

and more important

© 2015 Autodesk

Linguistic Quality Dashboard (dummy data)Performance Indicators, Comparisons and Trends

© 2015 Autodesk, Inc. All rights reserved.

Autodesk is a registered trademark of Autodesk, Inc., and/or its subsidiaries and/or affiliates in the USA and/or other countries. All other brand names, product names, or trademarks belong to their respective holders. Autodesk reserves the right to alter product and services offerings, and specifications and pricing at any time without notice, and is not responsible for typographical or graphical errors that may appear in this document.