PG 1 Caroline V Nelson Speech Solutions Team Technical Lead Nortel Enterprise Multimedia Professional Services August 20, 2007 Success Criteria for Speech Applications
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
PG 1
Caroline V NelsonSpeech Solutions Team Technical LeadNortel Enterprise Multimedia Professional ServicesAugust 20, 2007
Success Criteria for Speech Applications
NORTEL CONFIDENTIAL PG 2
Zen and the Art of Speech Applications
> We all want to deploy “successful” speech systems
> But what exactly is the definition of “success”?
> That depends on who you ask…
NORTEL CONFIDENTIAL PG 3
“Success” depends on who you ask…
Three different perspectives:
Purchasing Customer
Speech Scientist
End User (Caller)
NORTEL CONFIDENTIAL PG 4
Speech Scientist (designs/develops the app)
> The Micro View:
• Recognition Accuracy
• In/Out of Grammar Utterances
• Per-state “success” (retries, etc)
• Host database interaction
• Caller transactions (tasks across multiple dialog states)
NORTEL CONFIDENTIAL PG 5
“Micro” can’t see the forest for the trees…
> The recognition accuracy numbers look great!• (but there’s a very high rate of
callers speaking out-of-grammar…)
> The overall speech numbers look nice!• (but broken out into states shows
a hotspot down the flow path…)
> The speech is working like a charm!• (but the back-end host is
experiencing some serious issues…)
NORTEL CONFIDENTIAL PG 6
End Use Caller (calls into the app)
> The “Middle-Management” View:
• Task Completion
• Quick
• Easy
• “usability”
NORTEL CONFIDENTIAL PG 7
Now that we have the foundation…
> Need speech scientist metrics to achieve caller goals
• Callers prompted to speak in-grammar
• Utterances recognized• Error recovery through effective
strategies• Reliable back-end host data
> Caller “task” start & end points must be defined and agreed upon in advance to be properly logged.
> But if it’s awkward and cumbersome to use…
• Which is why we do usability testing – right?
NORTEL CONFIDENTIAL PG 8
Customer (purchases the app for business)
> The Macro View:
• Self-service call containment
• Cost reduction
• Customer (caller) satisfaction = loyalty/retention
> In the end it’s all about the bottom line $$$
NORTEL CONFIDENTIAL PG 9
Comparison is Key to “Macro”
> Usability testing for caller satisfaction• Benchmarking against “industry
standards”
> Need metrics from previous system (dtmf?) for comparison • To demonstrate level of
improvement
> If no previous system…• Then each self service completed
task is automation gain• Compare automation level
against benchmarks per industry
NORTEL CONFIDENTIAL PG 10
It’s all connected…
Macro
Micro
Mid-level
> Recognition accuracy
> In/Out of Grammar
> State success (retries, etc)
> Reliable host data
> Task completion
> Quick and easy
> “Usability”
> Customer satisfaction
> Self-service automation
> Call containment
> Bottom line $$$
NORTEL CONFIDENTIAL PG 11
Be careful what you measure…
> …because what you measure is what you’ll get.
> Metrics need to be identified, agreed upon and planned in advance – including any comparison measures.
Related Documents
