How to make sure the right quality is delivered by my translation vendor? (Sergio Pelino, Google)

Confidential & ProprietaryConfidential & Proprietary

TAUS QE Summit 2015Managing quality expectationsSergio Pelino - Google

Confidential & ProprietaryConfidential & Proprietary

Vendor management:● Vendor selection and SLAs (Quality, OTD, Cost KPIs)● 4 Translation vendors and 2 review vendors

Training/standards:● Training and onboarding programs● Language Services (LS) team owns quality standards, terminology and training

LQE and continuous feedback:● LQE: Automatic sampling of content for review based on historic performance and risk management

criteria● Continuous feedback: LQE tool allows to score quality and send immediate feedback for every

project to vendor/translators (pass/fail, err/1k, error types, full change report, reviewers comments)

Data management and sharing:● LQE data shared with all vendors (via API and UI dashboard)● KPIs tracking for immediate follow-ups/RCAs

How do we manage quality:

Confidential & ProprietaryConfidential & Proprietary

● Quality rules for reviewers can be overly complex and subjective which makes it difficult to have all reviewers aligned on same criteria when scoring.

● Working on getting quality at the start (context, product knowledge, great TMs and glossaries, QA and integrity checks in translation tool).

● Stretch target is to have all content on a minimum % sampling cadence (needed for data).

● Vendors capacity, talent management and retention, allocating the right resource to the content type.

Challenges:

Confidential & ProprietaryConfidential & Proprietary

Automatic, self-adjusting QE sampling logic based on:Vendor, Component, Language, Product

- Average errors/1k > 2- Pass rate > 95%- # of consecutive Passes > 3-6

Whitelisted Components:- Marketing- Legal

Sample 10% of projects

Send all to Review

Business rules*Projects

Vendor_1Help, JA, Android

Vendor_2UI, FR, Chrome

Vendor_3Marketing, IT, Adwords

Sample 20% of projects(y/n)

Send to review?

*Based on historic performance data on rolling window of previous 90 days

Confidential & ProprietaryConfidential & Proprietary

Our LQE tool

Chrome extension: runs on top of Google Translator ToolkitProduces instant feedback to Vendors and translators

LQE evaluation criteria

Pass/Fail: 3 error points/1,000 words

Confidential & ProprietaryConfidential & Proprietary

Instant, continuous feedback to vendor and translators

Confidential & ProprietaryConfidential & Proprietary

LQE Pass/Fail performance over time vs SLA (max fail rate = 5%)

Confidential & ProprietaryConfidential & Proprietary

Fail rate per language.

SLA: 5%

Confidential & ProprietaryConfidential & Proprietary

Vendor_x quality vs LSA

Confidential & ProprietaryConfidential & Proprietary

Vendor_x error distribution - identify areas for improvement

Confidential & ProprietaryConfidential & Proprietary

All vendors VS On Time delivery SLA (Max: 7%)

Confidential & ProprietaryConfidential & Proprietary

Vendors training tracking.

No translator should touch Google work without completing training.