Confidential & Proprietary Confidential & Proprietary TAUS QE Summit 2015 Managing quality expectations Sergio Pelino - Google
Apr 15, 2017
Confidential & ProprietaryConfidential & Proprietary
TAUS QE Summit 2015Managing quality expectationsSergio Pelino - Google
Confidential & ProprietaryConfidential & Proprietary
Vendor management:● Vendor selection and SLAs (Quality, OTD, Cost KPIs)● 4 Translation vendors and 2 review vendors
Training/standards:● Training and onboarding programs● Language Services (LS) team owns quality standards, terminology and training
LQE and continuous feedback:● LQE: Automatic sampling of content for review based on historic performance and risk management
criteria● Continuous feedback: LQE tool allows to score quality and send immediate feedback for every
project to vendor/translators (pass/fail, err/1k, error types, full change report, reviewers comments)
Data management and sharing:● LQE data shared with all vendors (via API and UI dashboard)● KPIs tracking for immediate follow-ups/RCAs
How do we manage quality:
Confidential & ProprietaryConfidential & Proprietary
● Quality rules for reviewers can be overly complex and subjective which makes it difficult to have all reviewers aligned on same criteria when scoring.
● Working on getting quality at the start (context, product knowledge, great TMs and glossaries, QA and integrity checks in translation tool).
● Stretch target is to have all content on a minimum % sampling cadence (needed for data).
● Vendors capacity, talent management and retention, allocating the right resource to the content type.
Challenges:
Confidential & ProprietaryConfidential & Proprietary
Automatic, self-adjusting QE sampling logic based on:Vendor, Component, Language, Product
- Average errors/1k > 2- Pass rate > 95%- # of consecutive Passes > 3-6
Whitelisted Components:- Marketing- Legal
Sample 10% of projects
Send all to Review
Business rules*Projects
Vendor_1Help, JA, Android
Vendor_2UI, FR, Chrome
Vendor_3Marketing, IT, Adwords
Sample 20% of projects(y/n)
Send to review?
*Based on historic performance data on rolling window of previous 90 days
Confidential & ProprietaryConfidential & Proprietary
Our LQE tool
Chrome extension: runs on top of Google Translator ToolkitProduces instant feedback to Vendors and translators
LQE evaluation criteria
Pass/Fail: 3 error points/1,000 words
Confidential & ProprietaryConfidential & Proprietary
Instant, continuous feedback to vendor and translators
Confidential & ProprietaryConfidential & Proprietary
LQE Pass/Fail performance over time vs SLA (max fail rate = 5%)
Confidential & ProprietaryConfidential & Proprietary
Fail rate per language.
SLA: 5%
Confidential & ProprietaryConfidential & Proprietary
Vendor_x quality vs LSA
Confidential & ProprietaryConfidential & Proprietary
Vendor_x error distribution - identify areas for improvement
Confidential & ProprietaryConfidential & Proprietary
All vendors VS On Time delivery SLA (Max: 7%)
Confidential & ProprietaryConfidential & Proprietary
Vendors training tracking.
No translator should touch Google work without completing training.