Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
Deliverable D6.6.1
QTLaunchPad Workshop:Quality Metrics for Human and Machine
Translation
Author(s): Stephen Doherty, Federico Gaspari, Josef van Genabith,Declan Groves, Ankit Srivastava (DCU)
Dissemination Level: Public
Date: 24.04.2013
1
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
Grant agreement no. 296347Project acronym QTLaunchPadProject full title Preparation and Launch of a Largescale Action for Quality
Translation TechnologyFunding scheme Coordination and Support ActionCoordinator Prof. Hans Uszkoreit (DFKI)Start date, duration 1 July 2012, 24 monthsDistribution PublicContractual date ofdelivery
March 2013 (Extended)
Actual date of delivery April 2013Deliverable number D6.6.1Deliverable title QTLaunchPad Workshop on Quality Metrics for Human and Machine
TranslationType ReportStatus and version DraftNumber of pagesContributing partners DCU, DFKI, USFDWP leader DCUTask leader DCUAuthors Stephen Doherty, Federico Gaspari, Josef van Genabith, Declan
Groves, Ankit SrivastavaEC project officer Kimmo RossiThe partners inQTLaunchPad are:
Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI),Germany
Dublin City University (DCU), Ireland
Institute for Language and Speech Processing, R.C. “Athena”(ILSP/ATHENA RC), Greece
The University of Sheffield (USFD), United Kingdom
For copies of reports, updates on project activities and other QTLaunchPadrelated information, contact:
DFKI GmbHQTLaunchPadDr. Aljoscha Burchardt [email protected]Moabit 91c Phone: +49 (30) 23895183810559 Berlin, Germany Fax: +49 (30) 238951810
Copies of reports and other material can also be accessed via http://www.qt21.eu/launchpad
© 2012, The Individual AuthorsNo part of this document may be reproduced or transmitted in any form, or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission from the copyright owner.
2
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
Table of Contents
1 Introduction1. 1 Multidimensional Quality Metrics1.2 MQM Workshop and Related Events
2 Workshop Feedback2.1 Principles & Dimensions2.2 Applications
3 Conclusion3.1 Summary3.2 Future Directions
AppendicesAppendix A Rome Workshop AgendaAppendix B Workshop A5 FlyersAppendix C Mailing List MaterialAppendix D Rome Workshop External ParticipantsAppendix E Miami Workshop GALA Participants
3
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
1 IntroductionThe aim of this report is to capture the discussions and feedback from the public workshop
on the QTLaunchPad quality metrics on human and machine translation. Following an
introduction to the multidimensional quality metrics (MQM) being developed in the project and
a description of the workshop format, this report:
1. provides specific feedback and discussion points for the MQM framework (Section 2);
2. highlights key commonalities (Section 2);
3. recommends future steps informed by the above within the project context (Section
3).
1. 1 Multidimensional Quality MetricsAs a core aspect of the QTLaunchPad project, the multidimensional quality metrics
framework (MQM) is a means of moving beyond the current shortcomings of existing
translation quality assessment. Quality assessment (QA) is an important task in the
translation workflow, especially in the context of machine translation. Traditional methods are
typically very subjective and involve assessors counting errors and their severity. This
approach has led to the formalisation of metrics for assigning errors to different types (e.g.
incorrect spelling, incorrect terminology, wrong verb agreement) and counting their
prevalence and severity in a random sample of translated content. This approach developed
into specifications such as the LISA QA Model and SAE J2450, however, these models have
not been updated consistently and have shortcomings in terms of validity and robustness.
The prevalence of the use of MT in translation and localization workshops also presents a
challenge for evaluation and QA processes due to the unique nature of MT output, and the
extensiveness of human intervention, e.g. use of raw MT output, human postedited MT. The
“onesizefitsall” approach of existing models simply cannot meet the needs and
expectations of a rapidly diversifying translation industry.
Following a systematic review of existing models and extensive public consultation, the
QTLaunchPad project is developing a new framework for assessing quality based on the
4
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
following principles:
Adaptability metrics must be adaptable to specific project types as projects, users,
requirements, and scenarios are unique. This allows for metrics to be tunable and
sensitive to each application;
Granularity metrics must allow for customizable degrees of granularity, from
extremely coarse to extremely fine, depending on the use case, e.g. evaluation of gist
translation visávis a detailed analysis to isolate errors;
Comparability results must be comparable and meaningful across jobs, projects,
users, and domains of application;
Suitability metrics must be suitable for both human and machine translation and
combinations thereof in addition to all technology and production profiles and users;
Standardised metrics must develop upon existing standards and established best
practice in order to optimise this progress while supporting interoperability, proprietary
methods, and customisable workflow integration;
Fairness existing metrics conflate errors in the source, errors in the target, and
actual translation errors. This results in problems identifying the cause of the problem,
where translators are often blamed. QA should be fair and recognise the work of
translators, especially when they add value in this way.
A more extensive and practical description, and further information on implementation, is
available from the project website , and in the publicly available deliverable D1.1.2: 1
Multidimensional Quality Metrics.
1.2 MQM Workshop and Related EventsAs a means of information gathering from relevant stakeholder groups, a survey was
conducted in late 2013 (whose results are documented in project deliverable D6.5.1: Report
on Requirements Gathering from Relevant Stakeholder Groups). Of 438 respondents across
approximately 40 countries, the survey examined a crosssection of the main stakeholder
groups targeted by the QTLaunchPad project: translators and LSPs, largescale public users
of language technology (LT), providers of LT, and corporate users.
The findings of the survey highlighted current trends and best practices, and were particularly
1 http://www.qt21.eu/launchpad/content/multidimensionalqualitymetrics
5
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
insightful regarding the utilisation of language and translation technologies in the context of
quality assessment and MT postediting. The survey also identified shortcomings, for
instance, in the absence of comparable and robust QA methods and in the quality thresholds
of current approaches to MT.
Informed by these findings (D6.5.1), several general topics of importance were identified and
appear below as ranked by workshop participants prior to the event :2
the need to move beyond current approaches to evaluation;
the availability of highquality corpora, datasets, systems, etc.;
funding for language and translation technologies;
the need for specialised knowledge and expertise;
the quality of current corpora, datasets, etc.;
uncertainty about the future of MT.
The project consortium identified colocation with the MultilingualWeb workshop in Rome
(March 14th, 2013) to be the ideal first venue for the QTLaunchPad MQM framework
presentation due to its location, resource cost, and audience (attendance of 150). The W3C
MultilingualWeb workshops are free and open to the public and concern standardisation and 3
best practice for multilingual and multicultural webbased information, e.g. localisation, code
standards. In sponsorship of the event, QTLaunchPad received email access via the
organiser to the attendees, a fullpage promotional piece for the project, and space for
exhibition in the main room of the event (pullup poster and flyers).
The final agenda for the Rome workshop can be found in Appendix A. The MQM workshop
took place in the morning of the fullday session, where the afternoon was assigned to the
research innovation application scenarios, RIAS, as detailed in D6.7.1. Each of the
aforementioned stakeholder groups were well represented at the workshop: 7 LSPs, 2 LT
providers, 2 corporate users, and 7 largescale public users. The structure of the MQM 4
workshop is outlined below and corresponds to the structure of feedback in Section 2 of this
report:
2
http://docs.google.com/spreadsheet/ccc?key=0AotdkIT3g3R8dEZsa2pVVElhQk5QT3FBZkRlT3Vkc1E&usp=sharing3 http://www.multilingualweb.eu/4 See Appendix D names and companies will be removed from the public version of this deliverable.
6
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
1. Overview of Metrics:
a. Principles
b. Issue Types
c. Dimensions
2. Using Dimensions to Build TaskRelevant Metrics
3. Demonstration of MQM Tool
4. Closing Discussion
In addition, QTLaunchPad ran several events at the Globalization and Localization
Association conference in Miami (March 17th20th with an attendance of approximately 250 5
delegates with QTLaunchPad flyers in each bag) supported by the project’s subcontractor
GALA. GALA is the world's largest association for the language industry, including translation
and localization. It is a nonprofit in nature and provides resources to thousands of its
members. The QTLaunchPad presence consisted of:
1. a 45minute talk on MQM as part of the main conference programme (43 attendees in
threeway parallel slot);
2. an interactive exhibition open through the conference;
3. a roundtable discussion on MQM (two 90minute slots).
The exhibition space made use of the same marketing materials as the Rome event: flyers
detailing the MQM, and a pullup banner. The combination of events allowed project
representatives many opportunities to present QTLaunchPad, the RIAS, and metrics to this
audience, which resulted in feedback and contact points for many aspects of the project
work, including feedback on RIAS as outlined in this report, the multidimensional quality
metrics (D6.6.1), and future directions (D6.8.1 and D6.8.2).
Promotional material for these events can be found on the QTLaunchPad website , the 6
MultilingualWeb website , the workshop flyer , email and social media announcements , and 7 8 9
on the GALA website . In addition to these channels, the workshop was advertised via email 10
5 http://www.galaglobal.org/conference/6
http://www.qt21.eu/launchpad/content/workshopresearchinnovationapplicationscenarios%E2%80%93romemarch14th20137 http://www.multilingualweb.eu/documents/romeworkshop/romeprogram8 See Appendix B9 See Appendix C10 http://www.galaglobal.org/conference/qtlaunchpadshowcase
7
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
lists from project partners, including CNGL, MTList, and social media (e.g. Twitter and 11
LinkedIn ).12
Prior to the above events, QTLaunchPad also ran a webinar via GALA to showcase the 13
MQM framework (February 21st, 2013). Following the presentation there was time for
questions and discussion from the 75 participants (out of a total of 162 registered), and
additional followups via the project’s LinkedIn page and optin email list .14
11 http://twitter.com/qtlaunchpad12 http://www.linkedin.com/groups?gid=480751813 A recording of the webinar can be found here:http://www.galaglobal.org/recordingspastwebinars#GSI and a PDF of the content here:http://www.galaglobal.org/files/21Feb2013_QTLaunchpad_PPT.pdf14 http://www.dfki.de/mailman/cgibin/listinfo/qtlpnews%20
8
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
2 Workshop FeedbackThis section describes the feedback and discussions from the above workshop and related
events into the categories of: metrics and applications.
2.1 Metrics While it’s not always possible, it’s important to know the contexts in which metrics are
used. This includes knowledge of: users, domains, formats, tools, resources, etc.
Taking source text quality in a translation metric into account is a positive
development, but may not be immediately possible in existing workflows.
The range of issue types in the MQM needs to be practical and contain customisable
degrees of granularity. It is perhaps not possible to find a balance that suits all users,
so some level of customisation is necessary, e.g. Canadian government’s SEPT
error categories contains approximately 700 aspects, while very comprehensive it is
not applicable in industrial application.
There is also a need to look at evaluation processes, not just at error metrics. TAUS
has a Dashboard that gives options for job types, end use, and tools, etc., but it fails
to address how the evaluation data are used and if they are meaningful and effective
or not.
Further to this, issues of comparability across jobs, projects, and evaluation
paradigms is burdensome. Metrics such as BLEU have become the de facto
standard in research, yet may not be at all meaningful for translators or buyers MQM
must work with the existing evaluation landscape.
Further attention should be given to the nature of error categories rather than errors
themselves; this may be fruitful in terms of preprocessing, standardisation, and
overall resource saving.
There will need to be welldocumented and easily accessible and understandable
content for the basis of the MQM, its usage, and its value over existing approaches.
Sensitivity to the different types of errors introduced by human translation, machine
translation, and combined approaches.
There should be a balance of preventative and reactive strategies for QA human in
the loop, rather than effort wasted, e.g. postedited content and evaluation data to be
9
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
used to improve MT processes MQM needs to include such a balance.
Finally, the validation of the MQM is critical. To learn from existing models, there
needs to be strict reliability and validity testing, etc.
2.2 Applications
It is well recognised that current models do not meet industry needs. It is apparent to
most stakeholders that there is a need for updating existing approaches to evaluation
and QA rather than following the status quo in a haphazard, internallyfocused, and
reactive way.
Translators are typically not consulted in the development of metrics and their usage.
Inclusion of these groups will be valuable, especially in the proposition of the MQM
that translations are penalised for correction errors in the source.
Evaluators are busy and sometimes even ‘lazy’. Despite an extensive list of errors,
the same small number of metrics are used from a dropdown menu.
The ambiguity of certain tools or types (of errors) can be a source of the previous
point, but more generally, can lead to poor evaluation results that can vary greatly
from person to person. To combat this, ambiguity should be reduced by using clearly
defined types and procedures with cooperation between the tool and the users. (to
suit their needs).
It is also ambiguous what is meant by compatibility in MQM. Such ubiquitous
compatibility with existing models is a claim that may be difficult to accomplish in
practice.
In terms of formats: what will MQM be compatible with, what options are there with
regard to input/process/output formats and encoding?
The evaluation/QA data from MQM should also be meaningful, customisable, and
allow for different degrees of granularity, otherwise the best features of the metrics
may be lost or ignored postevaluation.
Alongside MQM, the approach of avoiding errors in the first place should be pushed
preventative rather than reactive steps. Thus, standardisation and preprocessing are
necessary components to highquality translation results. Translation quality is linked
directly to pricing higher quality equates to higher compensation. It is therefore
important to establish thresholds, just like production workflows.
10
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
In terms of usage, how can issues with representative sampling be addressed in
MQM? Currently, there is a need to sample and assess in QA models, while AEMs
like BLEU can assess the whole document/system.
There needs to be a clear way for users of MQM and the tool to give feedback and be
more interactive with the project mailing lists and oneway communications are not
sufficient.
Open field testing will allow for the inclusion of as many viewpoints as possible for
different users in a variety of scenarios.
11
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
3 Conclusion
3.1 SummaryIn summary, the focus of stakeholders and users is apparent throughout the workshop
discussion and feedback. Stakeholder/user inclusion and buyin are pivotal in the uptake of
new tools such as the MQM. Therefore, the entry level for its use must be low, with support
and accessible materials from the onset. The value of such adoption must be clear and
quantifiable for it to be successful. The more specific feedback relating to the metrics will be
addressed in the update to the first version of the MQM; the second, revised version will be
available in project deliverable D1.1.2.
Further to the acquisition of this valuable discussion and feedback outlined in the previous
section, the workshops resulted in quantifiable gains in terms of QTLaunchPad’s public
exposure (traffic to the project homepage, www.qt21.eu, increased from 302 unique visits in
the month of February to 524 in March, and 168 in April ), awareness raising of RIAS topics, 15
and membership to the project in terms of social media (e.g. increase to 71 members on
LinkedIn ), mailing list (increase to 343 new optin members), and individual contacts with 16
stakeholders.
3.2 Future DirectionsIn addition to the above, future directions for the MQM were invited where the following points
were discussed:
The project’s critical mass and development of the MQM represent sufficient
resources for largescale field testing and further refinement with community/industry
input.
There needs to be an ongoing collaborative process where quantifiable value is
evident for all parties, especially from industry buyin viewpoint. In the context of MQM,
this may result in collaborative efforts to test the framework in a variety of realworld
contexts where feedback from users is paramount.
15 As of April 15th to be updated at month end.16 For comparison, the META LinkedIn group has 356 members.
12
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
The rollout of the MQM platform, translate5 , needs to be coordinate effectively and 17
usercentric to ensure the greatest uptake and effective incorporation of feedback.
This could come in the form of usability, feedback sidebar, twoway communication
between users and QTLaunchPad, education and educational materials,
incentivisation for users to give feedback on errors, crashes, etc.
Feedback should be documented and the improvement of the MQM with this value
input should be made clear to support ownership and adoption of MQM by the user
community.
17 http://www.translate5.net/
13
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
Appendices
14
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
Appendix A Rome Workshop Agenda
15
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
16
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
Appendix B Workshop A5 Flyers
17
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
18
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
19
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
Appendix C Mailing List MaterialMessage Begins
***
QTLaunchPad Workshops at MultilingualWeb, Rome
Dear Colleagues,
I wish to invite you to a workshop hosted by the ECfunded QTLaunchPad project on multidimensional quality metrics (MQM), and on usecases for a largescale future MT research initiative (RIAS), colocated with MultilingualWeb W3C in Rome, Italy.
Workshop Dates: March 14th, 2013Time: MQM 09:00 12:00 (lunch included for both workshops); RIAS 13:00–16:00Venue: Headquarters of the United Nations Food and Agriculture Organisation, Rome, ItalyCost: Free
1. Workshop on Multidimensional Quality Metrics (MQM)Translation Quality Assessment (TQA) has recently emerged as an important
business topic where formal metrics such as the LISA QA Model and SAE J2450 for human translation have helped, but automatic metrics for machine translation are currently suitable only for research projects, not for production environments. QTLaunchPad has developed a unified multidimensional framework for TQA that is built around quality metrics that move beyond the limitations of existing models and focus on richness and compatibility with usability as a core feature.
This workshop focuses on the measurement of translation quality. It introduces attendees to the metrics introduced above. It will demonstrate tools for creating project typespecific metrics and ensuring their validity for actual production tasks. Participants will further be invited to provide feedback and to discuss their own quality requirements and needs to help improve and further develop the model in a discussionoriented exploration of key issues related to TQA.
Click here to attend and find out more about this workshop
2. Workshop on Research Innovation Application Scenarios (RIAS)A central aim of QTLaunchPad is the preparation for a largescale research and
innovation action (QT21) in the application of research into of several core areas which
20
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
have been identified in close consultation with stakeholders in research and industry. These research innovation application scenarios or RIAS represent promising combinations of tasks, domains, users, industrial actors, demonstrators, innovation mechanisms, data, etc. Current suggestions under discussion include:
Automotive: technical documentation with the end user in mind, andinternal communication for multilingual environments;
Medical: highquality medical information for the general public, andemergency warnings (e.g. earthquakes) when multilingual data (e.g.SMS) need to be disseminated quickly and accurately;
Public: public consultations and information; Media: subtitling/audio descriptors, e.g. for lectures and persontoperson
communication; Language Learning: multimodal communications a foreign language
that is mastered only partially.
This workshop presents the progress of the exploration of these areas to participants and invites interactive discussions where attendees can add their own needs and requirements and provide welcomed feedback to the work carried out so far.
Click here to attend and find out more about this workshop
Thanks and best wishes on behalf of the QTLaunchPad team,
Stephen
***Message Ends
21
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
Appendix D Rome Workshop External Participants
Name Organisation Stakeholder Group
Stefania Cavallo University of Bologna LT Provider 1
Joanna Drugan University of East Anglia LT Provider 2
Marcello Federico FBK Public User 1
Jan Hajic UFAL Public User 2
David Lewis Trinity College Dublin & CNGL Public User 3
Bettina Lieske SAP AG Corporate User 1
Christian Lieske SAP AG Corporate User 2
Joseph Mariani LIMSI Public User 4
Isabella Massardo The Big Wave LSP 1
Radina Matic tcloc.com LSP 2
Luigi Muzii sQuid LSP 3
Alessandra Muzzi AMTrad Services LSP 4
Phil Ritchie VistaTEC LSP 5
Felix Sasaki DFKI Public User 5
Marco Scarpa Milkshake Studio SRL LSP 6
Volker Steinbiss University of Aachen Public User 6
Marco Trombetti Translated.net LSP 7
Feiyu Xu DFKI Public User 7
22
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
Appendix E Miami Workshop GALA Participants
GALA 2013 Miami Participant List
Tea Dietterich Director 2M Language Services Brisbane, Australia
Peter Redmond Senior Project Manager Ad Astra Washington DC, United States
Lena Toolsie President and CEO Ad Astra Washington DC, United States
Maxwell Hoffmann Global Product Evangelist, Tech Comm ADOBE SYSTEMS Beaverton, OR, UnitedStates
Quynn Le International Program Manager Adobe Systems, Incorporated Houston, United States
Vitaly Pedchenko Deputy Managing Director Advanced International Translations Kiev, Ukraine
Aleksandr Diatlov Head of Technical Support Advanced International Translations Kiev, Ukraine
Vladimir Pedchenko Managing Director Advanced International Translations Kiev, Ukraine
Katalin Varga managing director Afford Translations and Interpreting Ltd. Budapest, Hungary
Christopher Carter General Manager aLanguageBank New York, United States
Dimitra Hengen President Alpha Omega Translations Alexandria, VA 22306, United States
Dorothee Racette President American Translators Association Saranac, United States
George Chew Sr. Localization Specialist Amway Ada, Michigan, United States
Catherine Guinon Manager, Digital Operations Amway Corporation Ada, United States
Jaroslav Rázek Managing Director Aploq Ltd. Wroclaw, Poland
Mohamed Hassan Business Development Manager Arabize Cairo, Egypt
Roberto Ganzerli CEO Arancho Doc Bologna, Italy
David Boe Business Development Manager, North America Atril Solutions Portland, United States
Blandine Loze Managing Director Atril Solutions Paris, France
Joël Sigling Director AVB Language Services Amstelveen, Netherlands
Marie Flacassier COO BeatBabel San Diego, United States
Burckhardt Rueffer CEO BeatBabel San Diego, United States
Daniela Schobert Partner Blue Dimension, Lda Lisboa, Portugal
Cristina Gil Partner Blue Dimension, Lda Lisboa, Portugal
Tanushree Gupta General Manager, Production Braahmam Net Solutions Pvt. Ltd. DelhiNCR, India
Anjula Jyala Assistant Manager, Business Development Braahmam Net Solutions Pvt. Ltd. DelhiNCR,India
Anna Martirosian Resourcing and Operations Branded Translations Henderson, United States
Alan Melby Professor; President Brigham Young University and LTAC Provo, United States
Fabiano Cid Managing Director Ccaps Rio de Janeiro, Brazil
David Lewis Research Assistant Professor Centre for Next Generation Localisation Dublin, Ireland
Dominic Jones Research Assistant Centre for Next Generation Localisation Dublin, Ireland
Jiri Stejskal President & CEO CETRA Language Solutions Elins Park, United States
Tommy Tomolonis Project Manager CETRA Language Solutions Philadelphia, United States
23
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
Yu Li Senior Developer ClayTablet Technologies Toronto, Canada
Páraic Sheridan Associate Director CNGL Dublin, Ireland
Steve Gotz Commercial Development Manager CNGL Dublin, Ireland
Don DePalma Chief Strategy Officer Common Sense Advisory, Inc. Cambridge, United States
Miklos Urban General Manager Consell Translations Budapest, Hungary
Thomas Dimmer Business Development Manager CONTRAD Olsztyn, Poland
Marek Makosiej Managing Director CONTRAD Weranis i Makosiej Sp. J. Olsztyn, Poland
Gary Muddyman CEO/ Managing Director Conversis Global Oxford, United Kingdom
Briana McCrory Marketing Manager Conversis Global Oxford, United Kingdom
Tenesoya Pawlowsky Managing Director CPSL Barcelona, Spain
Livia Florensa Managing Director CPSL Madrid, Spain
Judy Crittenden Production Manager CTS LanguageLink Vancouver, United States
Angelica Perez Partner Delsurtranslations Córdoba, Argentina
Virginia Minhondo Partner Delsurtranslations Córdoba, Argentina
Arle Lommel Senior Consultant, Language Technology Lab DFKI Berlin, Germany
Andrew Lawless President DigIT / Rockant, Inc. Washington, D.C., United States
Asta Rusakevičienė CEO Diskusija UAB Vilnius, Lithuania
David Orban CEO Dotsub New York, United States
Michael Smolens Founder & Chairman Dotsub New York, United States
Marcel Bregman Sr. Localization Engineering Manager eBay Inc. San Jose, United States
Emily Shearer Director, Client Services Echo International Pittsburgh, United States
John Watkins President ENLASO Boise, ID, United States
Vigdis Eriksen President & CEO Eriksen Translations/Red T Brooklyn, United States
Jörgen Danielsen Managing Director Eule Lokalisierung GmbH Kiel, Germany
Mark Evenepoel CEO euroscript International S.A. Bertrange, Luxembourg
Carlos Hesano COO Executive Language Services Bloomfield Hills, United States
Sunil Kulkarni President Fidel Localization Services Tokyo, Japan
Karin Albiez German translator and reviewer Freelancer Barcelona, Spain
Saoussane Mrini Business Development Future Trans Ltd Cairo, Egypt
John Labati Chief Operating Officer G3 Translate New York, United States
Emre Akkas Chief Localizer Globalme Language&Technology Vancouver, Canada
Paula Ferrari Managing Director go global! Buenos Aires, Argentina
Chenhui Chu Student Graduate School of Informatics, Kyoto U. Kyoto, Japan
Aida Martirosyan Managing Director Haymillian London, United Kingdom
Miguel Martinez Transcreation Director Hogarth Worldwide London, United Kingdom
Meritxell Guitart President, Hogarth Americas Hogarth Worldwide Inc. New York, United States
Xavier Maza Cid Language Services Manager iDISC Barcelona, Spain
Pere GrivéAyguadé General Manager iDISC Barcelona, Spain
Olga López Project Manager iDISC Barcelona, Spain
Terena Bell CEO In Every Language Louisville, United States
24
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
Loic Dufresne De Virel Localization Strategist Intel Corp Portland, OR, United States
Rosana Bailone Director Interaction Translation Services Villa Mercdes, Argentina
William Skinner Chief Translator International Monetary Fund Washington, DC, United States
Beate Birkefeld Director of Translation Services Interpreters and Translators, Inc. (iTi) Manchester,United States
Ralph Strozza CEO Interpro Translation Solutions Lisle, United States
Mitch Donaldson Project Manager Interpro Translation Solutions, Inc. Lisle, United States
Manfred Schmitz Member, Executive Board Intertext Fremdsprachendienst e.G. Berlin, Germany
Olga Fomenko Marketing and PR Manager InText Translation Dnipropetrovsk, Ukraine
Céline Imbaud General Manager ITC GLOBAL TRANSLATIONS Jupiter, United States
Yaroslav Tugarev Business Development Manager Janus Moscow, Russia
William Rivers Executive Director JNCL / NCIS Washington, DC, United States
Crystal Goldie Policy Analyst JNCL / NCIS Washington D.C., United States
Miriam Valova Procurement Area Manager & Team Coord. Jonckers Translation & Engineering Brno,Czech Republic
Sergio Ruffolo COO JR Language Services Webster, NY, United States
Jackie Ruffolo President JR Language Services Webster, New York, United States
Patricia Bown Director of Sales, Americas Kilgray Translation Technologies Austin, Hungary
Peter Yoon CEO Korean LSP Oakland, United States
Mário Rodrigues Operations Director and Business Development L10N Studio Lisbon, Portugal
Zhang Steven Managing Director Landelion Communications Beijing, China
Starla Keith Executive Vice President Language Services Associates, Inc. Horsham, United States
Scott Ludwigsen President, Translation and Localization Division LanguageLine Solutions Portland, OR,United States
Antonio Nebres Director of TM and Localization Engineering LanguageWorkd New York, United States
Fernando D'Aloia Managing Director LATINDATA Buenos Aires, Argentina
Carlos Anllo Project Manager LatinLingua Buenos Aires, Argentina
Ariel Meyer Chief Business Officer, General Manager LatinLingua Miami, United States
Ignacio Valencia Rú Chief Orchestration Officer LatinLingua Buenos Aires, Argentina
Steve Yoon CEO Latis Global Seoul, Korea
Edward Kim Project Manager Latis Global Communications Seoul, South Korea
Lori Thicke CEO LexWorks Vancouver, Canada
Susan Amarino President Liaison Multilingual Services, Inc. Denver, United States
Jerzy Nedoma CEO LIDOLANG Technical Translations Kraków, Poland
Pablo Badia Mas Production Director Linguaserve I.S. S.A. Madrid, Spain
Pedro DiezOrzas CEO Linguaserve I.S. S.A. Madrid, Spain
Martin Roberts President Linguistic Systems, Inc. Cambridge, United States
Peter Basile Director of eCommerce Lionbridge International Boston, United States
Catalina Savill Customer Success Manager Lionbridge International Boston, United States
Gisela Donnarumma Customer Success Manager Lionbridge International Dublin, Ireland
Richard Sikes Principal Consultant Localization Flow Technologies Toronto, Canada
25
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
Aki Ito President LocalizationGuy Dallas, United States
Lilian Alves Mautone Managing Partner LocHouse Translation and DTP Rio de Janeiro, Brazil
Serge Gladkoff President Logrus International Philadelphia/Moscow, United States
Luciana Ramos Professional Translator and Interpreter Luciana Ramos Translation and TrainingRoldán, Argentina
Nima Nourkami Director Sales & Marketing Lucy Software and Services Waibstadt, Germany
Daniel Grasmick Managing Director Lucy Software and Services GmbH Waibstadt, Germany
Jennifer Brundage Senior consultant Language Tools Lucy Software and Services GmbH Waibstadt,Germany
Kadri Õismaa International Marketing Manager Luisa Translation Agency Tallinn, Estonia
Norman Newton Vice President ManpowerGroup Language Solutions Houston, United States
Norman Newton Vice President ManpowerGroup Language Solutions Houston, United States
Sacha O'Connell Doctoral Scholar Massey University Wellington, New Zealand
Richard Brooks Director, Strategy & Business Development MCIS Language Services Toronto, Canada
Josef Kubovsky Business Development Manager MemSource Prague, Czech Republic
David Canek CEO MemSource Prague, Czech Republic
ADELAIDA MAIDAGAN GENERAL DIRECTOR MONDRAGON LINGUA S COOP. MONDRAGON,Spain
ANE RUIZ DE ZARATE Director Translation office Mondragon MONDRAGON LINGUA S COOP.MONDRAGON, Spain
HAMISI AMANIDOVE Business Development Manager North America MONDRAGON LINGUA SCOOP. MONDRAGON, United States
Arturo Quintero Member of the Board Moravia Worldwide Brno, Czech Republic
Steven McNair Director of Sales North America MultiCorpora R&D Inc. Montreal, Canada
Jeremy Coombs VP of Operations MultiLing Provo, United States
Qianqian Wang Marketing Representative Multilingual Technologies Inc. Santa Clara, United States
Barbara Burbach Sr Localization PM NetApp Sunnyvale, United States
Aleksander Honma DIRECTOR NETWIRE CAMPINAS, Brazil
Andrew Jones Manager Translation Dept Nikon Precision Inc Belmont, United StatesJill SmithAguirre Strategic Account Executive OmniLingua Worldwide, LLC Cedar Rapids, UnitedStates
Marcela Becerra Director of Localization Services Paragon Language Services, Inc. Los Angeles, UnitedStates
Marina Mintz VP, Business Development Paragon Language Services, Inc. Los Angeles, United States
JeanPierre Parenty CEO Parenty Reitmeier Inc. Winnipeg, Canada
Vitor Silva Office Manager PHILOS Comunicação Global, Lda Porto, Portugal
Sufian Reiter Vice President of Sales Plunet Berlin, Germany
Nancy Radloff Business Development Manager Plunet Berlin, Germany
Tom Hoar Managing Director Precision Translation Tools Co., Ltd. Bangkok, Thailand
Shelly Priebe CEO Priebe & Associates Austin, United States
Engel Abalos Project Manager PTSGI Taipei, Taiwan
James Chen Sales Manager PTSGI Taipei, Taiwan
26
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
Square Fang President PTSGI Taipei, Taiwan
Stephen Doherty Postdoctoral Researcher QTLP, CNGL, Dublin City University Dublin, Ireland
Maya Hess CEO Red T New York, United States
Andrea Nahrwold Relationship Manager Rheinschrift Übersetzungen Cologne, Germany
Susan Zalkow Team Lead Project Management Rheinschrift Übersetzungen Cologne, Germany
María Gabriela Morales Director Rosario Traducciones y Servicios S.A. Rosario, Argentina
Rebecca Metcalf Senior Project Manager Rubric Edinburgh, Scotland
Udi Hershkovich VP of Business Development Safaba Translation Solutions Pittsburgh, United States
Anu CarnegieBrown Operations Manager Sandberg Translation Partners Whiteley, United Kingdom
Jesper Sandberg Managing Director Sandberg Translation Partners Ltd Whiteley, United Kingdom
Marcus Danei Product Manager SAP St. Leon, Germany
Wafaa Mohiy General ManagerEgypt Saudisoft Co. Ltd Cairo, Egypt
David Ashton VP, Consulting Services SDL Boulder, United States
Andy Reid Product Marketing Manager SDL International Reading, United Kingdom
Misael Diaz North America Business Manager, LSP Division SDL International Waltham, United States
Vicenta Ten Soriano Regional Sales Director SDL International Maidenhead, United KingdomTomas Ezpeleta Business Development Manager SDL International Waltham, United States
Maria Azqueta Arizcun Production Manager SeproTec Multilingual Solutions Madrid, Spain
Mieko Kuroki Graphic Designer SmartDTP Arequipa, Peru
Evelyn Paredes Business Development Manager SmartDTP Arequipa, Peru
Matt DeLoca VP, Sales Smartling New York, United States
Jack Welde Founder & CEO Smartling New York, United States
Justin Dignelli Account Executive Smartling New York, United States
Margherita Martella Senior Lead Localization Manager Spil Games Hilversum, The Netherlands
Patricia Doest Peccia Localization Project Manager Spil Games Hilversum, Netherlands
Emilio Alesiani Operations Manager STEP.IN. S.r.l. Rome, Italy
Grant Straker CEO Straker Translations Auckland, New Zealand
Suzie Mays Technical Product Manager SuccessFactors/Jobs2Web Minnetonka, United States
Cecilia Iros CEO suma Cordoba, Argentina
Beatriz Bonnet President & CEO Syntes Language Group Centennial, United States
Diego Bartolomé CEO tauyou <language technology> Sabadell, Spain
Julie DiRosa International Project Manager tcworld GmbH Stuttgart, Germany
Michael Fritz CEO tcworld GmbH Stuttgart, Germany
Christian Hammer CEO Teknotrans AB Gothenburg, Sweden
Mauro Bertuol CEO TERRALINGUA Porto Alegre, Brazil
Juan Baquero CEI Texo SRL Córdoba, Argentina
Kim Harris Managing Director text & form Berlin, Germany
Horst Liebscher Director of Technology and Innovation text & form Berlin, Germany
Thomas Senf Managing Director text & form Berlin, Germany
Robert Etches CIO TextMinded Danmark Aarhus, Denmark
Raymond Monette VicePresident, Information Technology The LanguageWorks Inc. New York, United
27
Preparation and Launch of a Large-scale Action for Quality Translation Technology
D6.6.1: QTLaunchPad Workshop: Quality Metrics for Human and MachineTranslation
States
Alicia Assini Project Management Intern The Rosetta Foundation Dublin, Ireland
Jorge Del Rosario Project Manager Systems and Vendor Mgmt. The World Bank Washington, UnitedStatesIndra Samite Dir. Business Development TILDE RIga, Latvia
Stefan Gentz CEO TRACOM OHG Bonn, Germany
Laura Gori Owner TransEdit Group Srl Milan, Italy
Thomas Pennell Vice President TransPerfect Translations Inc New York, United States
Richard Estevez CEO Trusted Translations, Inc. Miami, United States
Gustavo Lucardi COO Trusted Translations, Inc. Miami, United States
Kimberly LeFrancois Account Executive TÜV SÜD America Inc. San Diego, United States
Kara Shannon Account Executive TÜV SÜD America Inc. San Diego, United States
Agustina Pioli Partner Two Ways Translation Services Cordoba, Argentina
Milena Rossi Partner Two Ways Translation Services Cordoba, Argentina
Reinhard Schäler Dirctor, LRC University of Limerick Limerick, Ireland
Lucia Specia Lecturer University of Sheffield Sheffield, United Kingdom
Agustina di Pratula Production Manager Ushuaia Solutions Rosario, Argentina
Kåre Lindahl CEO Venga Localization San Francisco, United States
Joseph Gomes Managing Director Viva Translations Lisbon, Portugal
Iris Capraro Galati Linguistic Manager Wells Fargo Bank Charlotte, United States
Olga Beregovaya VP, Language Tools Welocalize San Francisco, United States
Alex Yanishevsky senior solutions architect welocalize boston, United States
Robert Zessar Director of Market Strategy welocalize Frederick, United States
Gordon Husbands VP Sales & Marketing Wordbank London, United Kingdom
Kristyna Marrero Director of Sales & Marketing Wordfast New York, United States
Hannah CollyerBraham Operations Director World Writers London, United Kingdom
Brigt Skauge Transcreation Director World Writers London, United Kingdom
Véronique Özkaya Chief Sales and Marketing Officer Xplanation Leuven, Belgium
Bob Willans CEO XTM International Gerrards Cross, United Kingdom
Andrzej Zydron CTO XTM International Gerrards Cross, UK, United Kingdom
Bill Willette Sales Director XTM International Wells, United StatesTomasz Zalewski Key Account Manager XTRF Management Systems Krakow, Poland
Andrzej Nedoma Managing Director XTRF Management Systems Kraków, Poland
28