Deliverable D6.6.1 QTLaunchPad Workshop: Quality Metrics ... · audience, which resulted in feedback and contact points for many aspects of the project work, including feedback on

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

Deliverable D6.6.1

QTLaunchPad Workshop:Quality Metrics for Human and Machine

Translation

Author(s): Stephen Doherty, Federico Gaspari, Josef van Genabith,Declan Groves, Ankit Srivastava (DCU)

Dissemination Level: Public

Date: 24.04.2013

1

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

Grant agreement no. 296347Project acronym QTLaunchPadProject full title Preparation and Launch of a Large­scale Action for Quality

Translation TechnologyFunding scheme Coordination and Support ActionCoordinator Prof. Hans Uszkoreit (DFKI)Start date, duration 1 July 2012, 24 monthsDistribution PublicContractual date ofdelivery

March 2013 (Extended)

Actual date of delivery April 2013Deliverable number D6.6.1Deliverable title QTLaunchPad Workshop on Quality Metrics for Human and Machine

TranslationType ReportStatus and version DraftNumber of pagesContributing partners DCU, DFKI, USFDWP leader DCUTask leader DCUAuthors Stephen Doherty, Federico Gaspari, Josef van Genabith, Declan

Groves, Ankit SrivastavaEC project officer Kimmo RossiThe partners inQTLaunchPad are:

Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI),Germany

Dublin City University (DCU), Ireland

Institute for Language and Speech Processing, R.C. “Athena”(ILSP/ATHENA RC), Greece

The University of Sheffield (USFD), United Kingdom

For copies of reports, updates on project activities and other QTLaunchPad­related information, contact:

DFKI GmbHQTLaunchPadDr. Aljoscha Burchardt [email protected]­Moabit 91c Phone: +49 (30) 23895­183810559 Berlin, Germany Fax: +49 (30) 23895­1810

Copies of reports and other material can also be accessed via http://www.qt21.eu/launchpad

© 2012, The Individual AuthorsNo part of this document may be reproduced or transmitted in any form, or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission from the copyright owner.

2

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

Table of Contents

1 Introduction1. 1 Multidimensional Quality Metrics1.2 MQM Workshop and Related Events

2 Workshop Feedback2.1 Principles & Dimensions2.2 Applications

3 Conclusion3.1 Summary3.2 Future Directions

AppendicesAppendix A ­ Rome Workshop AgendaAppendix B ­ Workshop A5 FlyersAppendix C ­ Mailing List MaterialAppendix D ­ Rome Workshop External ParticipantsAppendix E ­ Miami Workshop GALA Participants

3

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

1 IntroductionThe aim of this report is to capture the discussions and feedback from the public workshop

on the QTLaunchPad quality metrics on human and machine translation. Following an

introduction to the multidimensional quality metrics (MQM) being developed in the project and

a description of the workshop format, this report:

1. provides specific feedback and discussion points for the MQM framework (Section 2);

2. highlights key commonalities (Section 2);

3. recommends future steps informed by the above within the project context (Section

3).

1. 1 Multidimensional Quality MetricsAs a core aspect of the QTLaunchPad project, the multidimensional quality metrics

framework (MQM) is a means of moving beyond the current shortcomings of existing

translation quality assessment. Quality assessment (QA) is an important task in the

translation workflow, especially in the context of machine translation. Traditional methods are

typically very subjective and involve assessors counting errors and their severity. This

approach has led to the formalisation of metrics for assigning errors to different types (e.g.

incorrect spelling, incorrect terminology, wrong verb agreement) and counting their

prevalence and severity in a random sample of translated content. This approach developed

into specifications such as the LISA QA Model and SAE J2450, however, these models have

not been updated consistently and have shortcomings in terms of validity and robustness.

The prevalence of the use of MT in translation and localization workshops also presents a

challenge for evaluation and QA processes due to the unique nature of MT output, and the

extensiveness of human intervention, e.g. use of raw MT output, human post­edited MT. The

“one­size­fits­all” approach of existing models simply cannot meet the needs and

expectations of a rapidly diversifying translation industry.

Following a systematic review of existing models and extensive public consultation, the

QTLaunchPad project is developing a new framework for assessing quality based on the

4

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

following principles:

Adaptability ­ metrics must be adaptable to specific project types as projects, users,

requirements, and scenarios are unique. This allows for metrics to be tunable and

sensitive to each application;

Granularity ­ metrics must allow for customizable degrees of granularity, from

extremely coarse to extremely fine, depending on the use case, e.g. evaluation of gist

translation vis­á­vis a detailed analysis to isolate errors;

Comparability ­ results must be comparable and meaningful across jobs, projects,

users, and domains of application;

Suitability ­ metrics must be suitable for both human and machine translation and

combinations thereof in addition to all technology and production profiles and users;

Standardised ­ metrics must develop upon existing standards and established best

practice in order to optimise this progress while supporting interoperability, proprietary

methods, and customisable workflow integration;

Fairness ­ existing metrics conflate errors in the source, errors in the target, and

actual translation errors. This results in problems identifying the cause of the problem,

where translators are often blamed. QA should be fair and recognise the work of

translators, especially when they add value in this way.

A more extensive and practical description, and further information on implementation, is

available from the project website , and in the publicly available deliverable D1.1.2: 1

Multidimensional Quality Metrics.

1.2 MQM Workshop and Related EventsAs a means of information gathering from relevant stakeholder groups, a survey was

conducted in late 2013 (whose results are documented in project deliverable D6.5.1: Report

on Requirements Gathering from Relevant Stakeholder Groups). Of 438 respondents across

approximately 40 countries, the survey examined a cross­section of the main stakeholder

groups targeted by the QTLaunchPad project: translators and LSPs, large­scale public users

of language technology (LT), providers of LT, and corporate users.

The findings of the survey highlighted current trends and best practices, and were particularly

1 http://www.qt21.eu/launchpad/content/multidimensional­quality­metrics

5

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

insightful regarding the utilisation of language and translation technologies in the context of

quality assessment and MT post­editing. The survey also identified shortcomings, for

instance, in the absence of comparable and robust QA methods and in the quality thresholds

of current approaches to MT.

Informed by these findings (D6.5.1), several general topics of importance were identified and

appear below as ranked by workshop participants prior to the event :2

the need to move beyond current approaches to evaluation;

the availability of high­quality corpora, datasets, systems, etc.;

funding for language and translation technologies;

the need for specialised knowledge and expertise;

the quality of current corpora, datasets, etc.;

uncertainty about the future of MT.

The project consortium identified co­location with the MultilingualWeb workshop in Rome

(March 14th, 2013) to be the ideal first venue for the QTLaunchPad MQM framework

presentation due to its location, resource cost, and audience (attendance of 150). The W3C

MultilingualWeb workshops are free and open to the public and concern standardisation and 3

best practice for multilingual and multicultural web­based information, e.g. localisation, code

standards. In sponsorship of the event, QTLaunchPad received e­mail access via the

organiser to the attendees, a full­page promotional piece for the project, and space for

exhibition in the main room of the event (pull­up poster and flyers).

The final agenda for the Rome workshop can be found in Appendix A. The MQM workshop

took place in the morning of the full­day session, where the afternoon was assigned to the

research innovation application scenarios, RIAS, as detailed in D6.7.1. Each of the

aforementioned stakeholder groups were well represented at the workshop: 7 LSPs, 2 LT

providers, 2 corporate users, and 7 large­scale public users. The structure of the MQM 4

workshop is outlined below and corresponds to the structure of feedback in Section 2 of this

report:

2

http://docs.google.com/spreadsheet/ccc?key=0AotdkIT3g3R8dEZsa2pVVElhQk5QT3FBZkRlT3Vkc1E&usp=sharing3 http://www.multilingualweb.eu/4 See Appendix D ­ names and companies will be removed from the public version of this deliverable.

6

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

1. Overview of Metrics:

a. Principles

b. Issue Types

c. Dimensions

2. Using Dimensions to Build Task­Relevant Metrics

3. Demonstration of MQM Tool

4. Closing Discussion

In addition, QTLaunchPad ran several events at the Globalization and Localization

Association conference in Miami (March 17th­20th with an attendance of approximately 250 5

delegates with QTLaunchPad flyers in each bag) supported by the project’s sub­contractor

GALA. GALA is the world's largest association for the language industry, including translation

and localization. It is a non­profit in nature and provides resources to thousands of its

members. The QTLaunchPad presence consisted of:

1. a 45­minute talk on MQM as part of the main conference programme (43 attendees in

three­way parallel slot);

2. an interactive exhibition open through the conference;

3. a roundtable discussion on MQM (two 90­minute slots).

The exhibition space made use of the same marketing materials as the Rome event: flyers

detailing the MQM, and a pull­up banner. The combination of events allowed project

representatives many opportunities to present QTLaunchPad, the RIAS, and metrics to this

audience, which resulted in feedback and contact points for many aspects of the project

work, including feedback on RIAS as outlined in this report, the multidimensional quality

metrics (D6.6.1), and future directions (D6.8.1 and D6.8.2).

Promotional material for these events can be found on the QTLaunchPad website , the 6

MultilingualWeb website , the workshop flyer , e­mail and social media announcements , and 7 8 9

on the GALA website . In addition to these channels, the workshop was advertised via e­mail 10

5 http://www.gala­global.org/conference/6

http://www.qt21.eu/launchpad/content/workshop­research­innovation­application­scenarios­%E2%80%93­rome­march­14th­20137 http://www.multilingualweb.eu/documents/rome­workshop/rome­program8 See Appendix B9 See Appendix C10 http://www.gala­global.org/conference/qtlaunchpad­showcase

7

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

lists from project partners, including CNGL, MT­List, and social media (e.g. Twitter and 11

LinkedIn ).12

Prior to the above events, QTLaunchPad also ran a webinar via GALA to showcase the 13

MQM framework (February 21st, 2013). Following the presentation there was time for

questions and discussion from the 75 participants (out of a total of 162 registered), and

additional follow­ups via the project’s LinkedIn page and opt­in e­mail list .14

11 http://twitter.com/qtlaunchpad12 http://www.linkedin.com/groups?gid=480751813 A recording of the webinar can be found here:http://www.gala­global.org/recordings­past­webinars#GSI and a PDF of the content here:http://www.gala­global.org/files/21Feb2013_QTLaunchpad_PPT.pdf14 http://www.dfki.de/mailman/cgi­bin/listinfo/qtlp­news%20

8

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

2 Workshop FeedbackThis section describes the feedback and discussions from the above workshop and related

events into the categories of: metrics and applications.

2.1 Metrics While it’s not always possible, it’s important to know the contexts in which metrics are

used. This includes knowledge of: users, domains, formats, tools, resources, etc.

Taking source text quality in a translation metric into account is a positive

development, but may not be immediately possible in existing workflows.

The range of issue types in the MQM needs to be practical and contain customisable

degrees of granularity. It is perhaps not possible to find a balance that suits all users,

so some level of customisation is necessary, e.g. Canadian government’s SEPT

error categories contains approximately 700 aspects, while very comprehensive it is

not applicable in industrial application.

There is also a need to look at evaluation processes, not just at error metrics. TAUS

has a Dashboard that gives options for job types, end use, and tools, etc., but it fails

to address how the evaluation data are used and if they are meaningful and effective

or not.

Further to this, issues of comparability across jobs, projects, and evaluation

paradigms is burdensome. Metrics such as BLEU have become the de facto

standard in research, yet may not be at all meaningful for translators or buyers ­ MQM

must work with the existing evaluation landscape.

Further attention should be given to the nature of error categories rather than errors

themselves; this may be fruitful in terms of pre­processing, standardisation, and

overall resource saving.

There will need to be well­documented and easily accessible and understandable

content for the basis of the MQM, its usage, and its value over existing approaches.

Sensitivity to the different types of errors introduced by human translation, machine

translation, and combined approaches.

There should be a balance of preventative and reactive strategies for QA ­ human in

the loop, rather than effort wasted, e.g. post­edited content and evaluation data to be

9

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

used to improve MT processes ­ MQM needs to include such a balance.

Finally, the validation of the MQM is critical. To learn from existing models, there

needs to be strict reliability and validity testing, etc.

2.2 Applications

It is well recognised that current models do not meet industry needs. It is apparent to

most stakeholders that there is a need for updating existing approaches to evaluation

and QA rather than following the status quo in a haphazard, internally­focused, and

reactive way.

Translators are typically not consulted in the development of metrics and their usage.

Inclusion of these groups will be valuable, especially in the proposition of the MQM

that translations are penalised for correction errors in the source.

Evaluators are busy and sometimes even ‘lazy’. Despite an extensive list of errors,

the same small number of metrics are used from a drop­down menu.

The ambiguity of certain tools or types (of errors) can be a source of the previous

point, but more generally, can lead to poor evaluation results that can vary greatly

from person to person. To combat this, ambiguity should be reduced by using clearly

defined types and procedures with cooperation between the tool and the users. (to

suit their needs).

It is also ambiguous what is meant by compatibility in MQM. Such ubiquitous

compatibility with existing models is a claim that may be difficult to accomplish in

practice.

In terms of formats: what will MQM be compatible with, what options are there with

regard to input/process/output formats and encoding?

The evaluation/QA data from MQM should also be meaningful, customisable, and

allow for different degrees of granularity, otherwise the best features of the metrics

may be lost or ignored post­evaluation.

Alongside MQM, the approach of avoiding errors in the first place should be pushed ­

preventative rather than reactive steps. Thus, standardisation and pre­processing are

necessary components to high­quality translation results. Translation quality is linked

directly to pricing ­ higher quality equates to higher compensation. It is therefore

important to establish thresholds, just like production workflows.

10

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

In terms of usage, how can issues with representative sampling be addressed in

MQM? Currently, there is a need to sample and assess in QA models, while AEMs

like BLEU can assess the whole document/system.

There needs to be a clear way for users of MQM and the tool to give feedback and be

more interactive with the project ­ mailing lists and one­way communications are not

sufficient.

Open field testing will allow for the inclusion of as many viewpoints as possible for

different users in a variety of scenarios.

11

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

3 Conclusion

3.1 SummaryIn summary, the focus of stakeholders and users is apparent throughout the workshop

discussion and feedback. Stakeholder/user inclusion and buy­in are pivotal in the uptake of

new tools such as the MQM. Therefore, the entry level for its use must be low, with support

and accessible materials from the onset. The value of such adoption must be clear and

quantifiable for it to be successful. The more specific feedback relating to the metrics will be

addressed in the update to the first version of the MQM; the second, revised version will be

available in project deliverable D1.1.2.

Further to the acquisition of this valuable discussion and feedback outlined in the previous

section, the workshops resulted in quantifiable gains in terms of QTLaunchPad’s public

exposure (traffic to the project homepage, www.qt21.eu, increased from 302 unique visits in

the month of February to 524 in March, and 168 in April ), awareness raising of RIAS topics, 15

and membership to the project in terms of social media (e.g. increase to 71 members on

LinkedIn ), mailing list (increase to 343 new opt­in members), and individual contacts with 16

stakeholders.

3.2 Future DirectionsIn addition to the above, future directions for the MQM were invited where the following points

were discussed:

The project’s critical mass and development of the MQM represent sufficient

resources for large­scale field testing and further refinement with community/industry

input.

There needs to be an ongoing collaborative process where quantifiable value is

evident for all parties, especially from industry buy­in viewpoint. In the context of MQM,

this may result in collaborative efforts to test the framework in a variety of real­world

contexts where feedback from users is paramount.

15 As of April 15th ­ to be updated at month end.16 For comparison, the META LinkedIn group has 356 members.

12

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

The roll­out of the MQM platform, translate5 , needs to be coordinate effectively and 17

user­centric to ensure the greatest uptake and effective incorporation of feedback.

This could come in the form of usability, feedback sidebar, two­way communication

between users and QTLaunchPad, education and educational materials,

incentivisation for users to give feedback on errors, crashes, etc.

Feedback should be documented and the improvement of the MQM with this value

input should be made clear to support ownership and adoption of MQM by the user

community.

17 http://www.translate5.net/

13

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

Appendices

14

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

Appendix A ­ Rome Workshop Agenda

15

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

16

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

Appendix B ­ Workshop A5 Flyers

17

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

18

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

19

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

Appendix C ­ Mailing List MaterialMessage Begins

***

QTLaunchPad Workshops at MultilingualWeb, Rome

Dear Colleagues,

I wish to invite you to a workshop hosted by the EC­funded QTLaunchPad project on multidimensional quality metrics (MQM), and on use­cases for a large­scale future MT research initiative (RIAS), co­located with MultilingualWeb W3C in Rome, Italy.

Workshop Dates: March 14th, 2013Time: MQM 09:00 ­ 12:00 (lunch included for both workshops); RIAS 13:00–16:00Venue: Headquarters of the United Nations Food and Agriculture Organisation, Rome, ItalyCost: Free

1. Workshop on Multidimensional Quality Metrics (MQM)Translation Quality Assessment (TQA) has recently emerged as an important

business topic where formal metrics such as the LISA QA Model and SAE J2450 for human translation have helped, but automatic metrics for machine translation are currently suitable only for research projects, not for production environments. QTLaunchPad has developed a unified multidimensional framework for TQA that is built around quality metrics that move beyond the limitations of existing models and focus on richness and compatibility with usability as a core feature.

This workshop focuses on the measurement of translation quality. It introduces attendees to the metrics introduced above. It will demonstrate tools for creating project type­specific metrics and ensuring their validity for actual production tasks. Participants will further be invited to provide feedback and to discuss their own quality requirements and needs to help improve and further develop the model in a discussion­oriented exploration of key issues related to TQA.

Click here to attend and find out more about this workshop

2. Workshop on Research Innovation Application Scenarios (RIAS)A central aim of QTLaunchPad is the preparation for a large­scale research and

innovation action (QT21) in the application of research into of several core areas which

20

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

have been identified in close consultation with stakeholders in research and industry. These research innovation application scenarios or RIAS represent promising combinations of tasks, domains, users, industrial actors, demonstrators, innovation mechanisms, data, etc. Current suggestions under discussion include:

Automotive: technical documentation with the end user in mind, andinternal communication for multilingual environments;

Medical: high­quality medical information for the general public, andemergency warnings (e.g. earthquakes) when multilingual data (e.g.SMS) need to be disseminated quickly and accurately;

Public: public consultations and information; Media: subtitling/audio descriptors, e.g. for lectures and person­to­person

communication; Language Learning: multi­modal communications a foreign language

that is mastered only partially.

This workshop presents the progress of the exploration of these areas to participants and invites interactive discussions where attendees can add their own needs and requirements and provide welcomed feedback to the work carried out so far.

Click here to attend and find out more about this workshop

Thanks and best wishes on behalf of the QTLaunchPad team,

Stephen

***Message Ends

21

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

Appendix D ­ Rome Workshop External Participants

Name Organisation Stakeholder Group

Stefania Cavallo University of Bologna LT Provider 1

Joanna Drugan University of East Anglia LT Provider 2

Marcello Federico FBK Public User 1

Jan Hajic UFAL Public User 2

David Lewis Trinity College Dublin & CNGL Public User 3

Bettina Lieske SAP AG Corporate User 1

Christian Lieske SAP AG Corporate User 2

Joseph Mariani LIMSI Public User 4

Isabella Massardo The Big Wave LSP 1

Radina Matic tcloc.com LSP 2

Luigi Muzii sQuid LSP 3

Alessandra Muzzi AMTrad Services LSP 4

Phil Ritchie VistaTEC LSP 5

Felix Sasaki DFKI Public User 5

Marco Scarpa Milkshake Studio SRL LSP 6

Volker Steinbiss University of Aachen Public User 6

Marco Trombetti Translated.net LSP 7

Feiyu Xu DFKI Public User 7

22

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

Appendix E ­ Miami Workshop GALA Participants

GALA 2013 Miami Participant List

Tea Dietterich Director 2M Language Services Brisbane, Australia

Peter Redmond Senior Project Manager Ad Astra Washington DC, United States

Lena Toolsie President and CEO Ad Astra Washington DC, United States

Maxwell Hoffmann Global Product Evangelist, Tech Comm ADOBE SYSTEMS Beaverton, OR, UnitedStates

Quynn Le International Program Manager Adobe Systems, Incorporated Houston, United States

Vitaly Pedchenko Deputy Managing Director Advanced International Translations Kiev, Ukraine

Aleksandr Diatlov Head of Technical Support Advanced International Translations Kiev, Ukraine

Vladimir Pedchenko Managing Director Advanced International Translations Kiev, Ukraine

Katalin Varga managing director Afford Translations and Interpreting Ltd. Budapest, Hungary

Christopher Carter General Manager aLanguageBank New York, United States

Dimitra Hengen President Alpha Omega Translations Alexandria, VA 22306, United States

Dorothee Racette President American Translators Association Saranac, United States

George Chew Sr. Localization Specialist Amway Ada, Michigan, United States

Catherine Guinon Manager, Digital Operations Amway Corporation Ada, United States

Jaroslav Rázek Managing Director Aploq Ltd. Wroclaw, Poland

Mohamed Hassan Business Development Manager Arabize Cairo, Egypt

Roberto Ganzerli CEO Arancho Doc Bologna, Italy

David Boe Business Development Manager, North America Atril Solutions Portland, United States

Blandine Loze Managing Director Atril Solutions Paris, France

Joël Sigling Director AVB Language Services Amstelveen, Netherlands

Marie Flacassier COO BeatBabel San Diego, United States

Burckhardt Rueffer CEO BeatBabel San Diego, United States

Daniela Schobert Partner Blue Dimension, Lda Lisboa, Portugal

Cristina Gil Partner Blue Dimension, Lda Lisboa, Portugal

Tanushree Gupta General Manager, Production Braahmam Net Solutions Pvt. Ltd. Delhi­NCR, India

Anjula Jyala Assistant Manager, Business Development Braahmam Net Solutions Pvt. Ltd. Delhi­NCR,India

Anna Martirosian Resourcing and Operations Branded Translations Henderson, United States

Alan Melby Professor; President Brigham Young University and LTAC Provo, United States

Fabiano Cid Managing Director Ccaps Rio de Janeiro, Brazil

David Lewis Research Assistant Professor Centre for Next Generation Localisation Dublin, Ireland

Dominic Jones Research Assistant Centre for Next Generation Localisation Dublin, Ireland

Jiri Stejskal President & CEO CETRA Language Solutions Elins Park, United States

Tommy Tomolonis Project Manager CETRA Language Solutions Philadelphia, United States

23

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

Yu Li Senior Developer ClayTablet Technologies Toronto, Canada

Páraic Sheridan Associate Director CNGL Dublin, Ireland

Steve Gotz Commercial Development Manager CNGL Dublin, Ireland

Don DePalma Chief Strategy Officer Common Sense Advisory, Inc. Cambridge, United States

Miklos Urban General Manager Consell Translations Budapest, Hungary

Thomas Dimmer Business Development Manager CONTRAD Olsztyn, Poland

Marek Makosiej Managing Director CONTRAD Weranis i Makosiej Sp. J. Olsztyn, Poland

Gary Muddyman CEO/ Managing Director Conversis Global Oxford, United Kingdom

Briana McCrory Marketing Manager Conversis Global Oxford, United Kingdom

Tenesoya Pawlowsky Managing Director CPSL Barcelona, Spain

Livia Florensa Managing Director CPSL Madrid, Spain

Judy Crittenden Production Manager CTS LanguageLink Vancouver, United States

Angelica Perez Partner Delsurtranslations Córdoba, Argentina

Virginia Minhondo Partner Delsurtranslations Córdoba, Argentina

Arle Lommel Senior Consultant, Language Technology Lab DFKI Berlin, Germany

Andrew Lawless President Dig­IT / Rockant, Inc. Washington, D.C., United States

Asta Rusakevičienė CEO Diskusija UAB Vilnius, Lithuania

David Orban CEO Dotsub New York, United States

Michael Smolens Founder & Chairman Dotsub New York, United States

Marcel Bregman Sr. Localization Engineering Manager eBay Inc. San Jose, United States

Emily Shearer Director, Client Services Echo International Pittsburgh, United States

John Watkins President ENLASO Boise, ID, United States

Vigdis Eriksen President & CEO Eriksen Translations/Red T Brooklyn, United States

Jörgen Danielsen Managing Director Eule Lokalisierung GmbH Kiel, Germany

Mark Evenepoel CEO euroscript International S.A. Bertrange, Luxembourg

Carlos Hesano COO Executive Language Services Bloomfield Hills, United States

Sunil Kulkarni President Fidel Localization Services Tokyo, Japan

Karin Albiez German translator and reviewer Freelancer Barcelona, Spain

Saoussane Mrini Business Development Future Trans Ltd Cairo, Egypt

John Labati Chief Operating Officer G3 Translate New York, United States

Emre Akkas Chief Localizer Globalme Language&Technology Vancouver, Canada

Paula Ferrari Managing Director go global! Buenos Aires, Argentina

Chenhui Chu Student Graduate School of Informatics, Kyoto U. Kyoto, Japan

Aida Martirosyan Managing Director Haymillian London, United Kingdom

Miguel Martinez Transcreation Director Hogarth Worldwide London, United Kingdom

Meritxell Guitart President, Hogarth Americas Hogarth Worldwide Inc. New York, United States

Xavier Maza Cid Language Services Manager iDISC Barcelona, Spain

Pere Grivé­Ayguadé General Manager iDISC Barcelona, Spain

Olga López Project Manager iDISC Barcelona, Spain

Terena Bell CEO In Every Language Louisville, United States

24

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

Loic Dufresne De Virel Localization Strategist Intel Corp Portland, OR, United States

Rosana Bailone Director Interaction Translation Services Villa Mercdes, Argentina

William Skinner Chief Translator International Monetary Fund Washington, DC, United States

Beate Birkefeld Director of Translation Services Interpreters and Translators, Inc. (iTi) Manchester,United States

Ralph Strozza CEO Interpro Translation Solutions Lisle, United States

Mitch Donaldson Project Manager Interpro Translation Solutions, Inc. Lisle, United States

Manfred Schmitz Member, Executive Board Intertext Fremdsprachendienst e.G. Berlin, Germany

Olga Fomenko Marketing and PR Manager InText Translation Dnipropetrovsk, Ukraine

Céline Imbaud General Manager ITC GLOBAL TRANSLATIONS Jupiter, United States

Yaroslav Tugarev Business Development Manager Janus Moscow, Russia

William Rivers Executive Director JNCL / NCIS Washington, DC, United States

Crystal Goldie Policy Analyst JNCL / NCIS Washington D.C., United States

Miriam Valova Procurement Area Manager & Team Coord. Jonckers Translation & Engineering Brno,Czech Republic

Sergio Ruffolo COO JR Language Services Webster, NY, United States

Jackie Ruffolo President JR Language Services Webster, New York, United States

Patricia Bown Director of Sales, Americas Kilgray Translation Technologies Austin, Hungary

Peter Yoon CEO Korean LSP Oakland, United States

Mário Rodrigues Operations Director and Business Development L10N Studio Lisbon, Portugal

Zhang Steven Managing Director Landelion Communications Beijing, China

Starla Keith Executive Vice President Language Services Associates, Inc. Horsham, United States

Scott Ludwigsen President, Translation and Localization Division LanguageLine Solutions Portland, OR,United States

Antonio Nebres Director of TM and Localization Engineering LanguageWorkd New York, United States

Fernando D'Aloia Managing Director LATIN­DATA Buenos Aires, Argentina

Carlos Anllo Project Manager LatinLingua Buenos Aires, Argentina

Ariel Meyer Chief Business Officer, General Manager LatinLingua Miami, United States

Ignacio Valencia Rú Chief Orchestration Officer LatinLingua Buenos Aires, Argentina

Steve Yoon CEO Latis Global Seoul, Korea

Edward Kim Project Manager Latis Global Communications Seoul, South Korea

Lori Thicke CEO LexWorks Vancouver, Canada

Susan Amarino President Liaison Multilingual Services, Inc. Denver, United States

Jerzy Nedoma CEO LIDO­LANG Technical Translations Kraków, Poland

Pablo Badia Mas Production Director Linguaserve I.S. S.A. Madrid, Spain

Pedro Diez­Orzas CEO Linguaserve I.S. S.A. Madrid, Spain

Martin Roberts President Linguistic Systems, Inc. Cambridge, United States

Peter Basile Director of eCommerce Lionbridge International Boston, United States

Catalina Savill Customer Success Manager Lionbridge International Boston, United States

Gisela Donnarumma Customer Success Manager Lionbridge International Dublin, Ireland

Richard Sikes Principal Consultant Localization Flow Technologies Toronto, Canada

25

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

Aki Ito President LocalizationGuy Dallas, United States

Lilian Alves Mautone Managing Partner LocHouse Translation and DTP Rio de Janeiro, Brazil

Serge Gladkoff President Logrus International Philadelphia/Moscow, United States

Luciana Ramos Professional Translator and Interpreter Luciana Ramos ­ Translation and TrainingRoldán, Argentina

Nima Nourkami Director Sales & Marketing Lucy Software and Services Waibstadt, Germany

Daniel Grasmick Managing Director Lucy Software and Services GmbH Waibstadt, Germany

Jennifer Brundage Senior consultant Language Tools Lucy Software and Services GmbH Waibstadt,Germany

Kadri Õismaa International Marketing Manager Luisa Translation Agency Tallinn, Estonia

Norman Newton Vice President ManpowerGroup Language Solutions Houston, United States

Norman Newton Vice President ManpowerGroup Language Solutions Houston, United States

Sacha O'Connell Doctoral Scholar Massey University Wellington, New Zealand

Richard Brooks Director, Strategy & Business Development MCIS Language Services Toronto, Canada

Josef Kubovsky Business Development Manager MemSource Prague, Czech Republic

David Canek CEO MemSource Prague, Czech Republic

ADELAIDA MAIDAGAN GENERAL DIRECTOR MONDRAGON LINGUA S COOP. MONDRAGON,Spain

ANE RUIZ DE ZARATE Director Translation office Mondragon MONDRAGON LINGUA S COOP.MONDRAGON, Spain

HAMISI AMANI­DOVE Business Development Manager North America MONDRAGON LINGUA SCOOP. MONDRAGON, United States

Arturo Quintero Member of the Board Moravia Worldwide Brno, Czech Republic

Steven McNair Director of Sales North America MultiCorpora R&D Inc. Montreal, Canada

Jeremy Coombs VP of Operations MultiLing Provo, United States

Qianqian Wang Marketing Representative Multilingual Technologies Inc. Santa Clara, United States

Barbara Burbach Sr Localization PM NetApp Sunnyvale, United States

Aleksander Honma DIRECTOR NETWIRE CAMPINAS, Brazil

Andrew Jones Manager ­ Translation Dept Nikon Precision Inc Belmont, United StatesJill Smith­Aguirre Strategic Account Executive OmniLingua Worldwide, LLC Cedar Rapids, UnitedStates

Marcela Becerra Director of Localization Services Paragon Language Services, Inc. Los Angeles, UnitedStates

Marina Mintz VP, Business Development Paragon Language Services, Inc. Los Angeles, United States

Jean­Pierre Parenty CEO Parenty Reitmeier Inc. Winnipeg, Canada

Vitor Silva Office Manager PHILOS ­ Comunicação Global, Lda Porto, Portugal

Sufian Reiter Vice President of Sales Plunet Berlin, Germany

Nancy Radloff Business Development Manager Plunet Berlin, Germany

Tom Hoar Managing Director Precision Translation Tools Co., Ltd. Bangkok, Thailand

Shelly Priebe CEO Priebe & Associates Austin, United States

Engel Abalos Project Manager PTSGI Taipei, Taiwan

James Chen Sales Manager PTSGI Taipei, Taiwan

26

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

Square Fang President PTSGI Taipei, Taiwan

Stephen Doherty Post­doctoral Researcher QTLP, CNGL, Dublin City University Dublin, Ireland

Maya Hess CEO Red T New York, United States

Andrea Nahrwold Relationship Manager Rheinschrift Übersetzungen Cologne, Germany

Susan Zalkow Team Lead Project Management Rheinschrift Übersetzungen Cologne, Germany

María Gabriela Morales Director Rosario Traducciones y Servicios S.A. Rosario, Argentina

Rebecca Metcalf Senior Project Manager Rubric Edinburgh, Scotland

Udi Hershkovich VP of Business Development Safaba Translation Solutions Pittsburgh, United States

Anu Carnegie­Brown Operations Manager Sandberg Translation Partners Whiteley, United Kingdom

Jesper Sandberg Managing Director Sandberg Translation Partners Ltd Whiteley, United Kingdom

Marcus Danei Product Manager SAP St. Leon, Germany

Wafaa Mohiy General Manager­Egypt Saudisoft Co. Ltd Cairo, Egypt

David Ashton VP, Consulting Services SDL Boulder, United States

Andy Reid Product Marketing Manager SDL International Reading, United Kingdom

Misael Diaz North America Business Manager, LSP Division SDL International Waltham, United States

Vicenta Ten Soriano Regional Sales Director SDL International Maidenhead, United KingdomTomas Ezpeleta Business Development Manager SDL International Waltham, United States

Maria Azqueta Arizcun Production Manager SeproTec Multilingual Solutions Madrid, Spain

Mieko Kuroki Graphic Designer SmartDTP Arequipa, Peru

Evelyn Paredes Business Development Manager SmartDTP Arequipa, Peru

Matt DeLoca VP, Sales Smartling New York, United States

Jack Welde Founder & CEO Smartling New York, United States

Justin Dignelli Account Executive Smartling New York, United States

Margherita Martella Senior Lead Localization Manager Spil Games Hilversum, The Netherlands

Patricia Doest ­ Peccia Localization Project Manager Spil Games Hilversum, Netherlands

Emilio Alesiani Operations Manager STEP.IN. S.r.l. Rome, Italy

Grant Straker CEO Straker Translations Auckland, New Zealand

Suzie Mays Technical Product Manager SuccessFactors/Jobs2Web Minnetonka, United States

Cecilia Iros CEO suma Cordoba, Argentina

Beatriz Bonnet President & CEO Syntes Language Group Centennial, United States

Diego Bartolomé CEO tauyou <language technology> Sabadell, Spain

Julie DiRosa International Project Manager tcworld GmbH Stuttgart, Germany

Michael Fritz CEO tcworld GmbH Stuttgart, Germany

Christian Hammer CEO Teknotrans AB Gothenburg, Sweden

Mauro Bertuol CEO TERRALINGUA Porto Alegre, Brazil

Juan Baquero CEI Texo SRL Córdoba, Argentina

Kim Harris Managing Director text & form Berlin, Germany

Horst Liebscher Director of Technology and Innovation text & form Berlin, Germany

Thomas Senf Managing Director text & form Berlin, Germany

Robert Etches CIO TextMinded Danmark Aarhus, Denmark

Raymond Monette Vice­President, Information Technology The LanguageWorks Inc. New York, United

27

Preparation and Launch of a Large-scale Action for Quality Translation Technology

D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation

States

Alicia Assini Project Management Intern The Rosetta Foundation Dublin, Ireland

Jorge Del Rosario Project Manager ­ Systems and Vendor Mgmt. The World Bank Washington, UnitedStatesIndra Samite Dir. Business Development TILDE RIga, Latvia

Stefan Gentz CEO TRACOM OHG Bonn, Germany

Laura Gori Owner Trans­Edit Group Srl Milan, Italy

Thomas Pennell Vice President TransPerfect Translations Inc New York, United States

Richard Estevez CEO Trusted Translations, Inc. Miami, United States

Gustavo Lucardi COO Trusted Translations, Inc. Miami, United States

Kimberly LeFrancois Account Executive TÜV SÜD America Inc. San Diego, United States

Kara Shannon Account Executive TÜV SÜD America Inc. San Diego, United States

Agustina Pioli Partner Two Ways Translation Services Cordoba, Argentina

Milena Rossi Partner Two Ways Translation Services Cordoba, Argentina

Reinhard Schäler Dirctor, LRC University of Limerick Limerick, Ireland

Lucia Specia Lecturer University of Sheffield Sheffield, United Kingdom

Agustina di Pratula Production Manager Ushuaia Solutions Rosario, Argentina

Kåre Lindahl CEO Venga Localization San Francisco, United States

Joseph Gomes Managing Director Viva Translations Lisbon, Portugal

Iris Capraro Galati Linguistic Manager Wells Fargo Bank Charlotte, United States

Olga Beregovaya VP, Language Tools Welocalize San Francisco, United States

Alex Yanishevsky senior solutions architect welocalize boston, United States

Robert Zessar Director of Market Strategy welocalize Frederick, United States

Gordon Husbands VP Sales & Marketing Wordbank London, United Kingdom

Kristyna Marrero Director of Sales & Marketing Wordfast New York, United States

Hannah Collyer­Braham Operations Director World Writers London, United Kingdom

Brigt Skauge Transcreation Director World Writers London, United Kingdom

Véronique Özkaya Chief Sales and Marketing Officer Xplanation Leuven, Belgium

Bob Willans CEO XTM International Gerrards Cross, United Kingdom

Andrzej Zydron CTO XTM International Gerrards Cross, UK, United Kingdom

Bill Willette Sales Director XTM International Wells, United StatesTomasz Zalewski Key Account Manager XTRF Management Systems Krakow, Poland

Andrzej Nedoma Managing Director XTRF Management Systems Kraków, Poland

28