Page 1
Collaborates with:•
Detecting Customer Complaint Escalation with Recurrent Neural Networks and Manually-Engineered Features
Image from:https://www.ratchetandwrench.com/articles/5045-dealing-with-customer-complaints
Wei Yang, Luchen Tan, Chunwei Lu, Anqi Cui, Han Li, Xi Chen, Kun Xiong, Muzi Wang, Ming Li, Jian Pei and Jimmy Lin
Page 2
Who are we?
RSVP.ai is a Canadian startup based in Waterloo, Ontario
that aims to build deep natural language understanding
systems to facilitate seamless dialogues between humans
and machines.
2
One of the largest Chinese e-commerce company. As of the first quarter of 2018, its platform has
301.8 million active users.
Page 3
!3
Hi, Agent ID 1234 from JD.com, happy to assist
you!
I’ve told you the address SEVERAL TIMES already!!! But you’ve wasted my time
by making me go back to the original address.
Hi, can you please provide me with the order number?
I’ll look into this for you!
The same problem has happened on many orders. I want to protect
my rights as a consumer!!!
Page 4
!4Image from: http://www.customerexperienceinsight.com/customers-said-you-suck-handling-complaints/
Page 5
Customer Complaint Escalation
!5
Customers
Tough-to-Please Customers
Company’s Agents
Consumer Protection Bureau
~70%
~30%
~75%: Text ~25%: Others
Page 6
Detecting Customer Complaint Escalation• 300+ complaints every day!
• Bad customer service experience causes serious brand damage to JD.com.
• Importance of real time detection system
!6
Online classification problem over dialogue
Hard problem! <0.01% complaints!1 week to 1 month13%
1 day to 1 week41%
<1 day47%
How long does it takes for the customer complaints after they talk to the agents?
Page 7
Model
!7
Tf-idf vectors
Manually-engineered featuresNeural Network!
Page 8
Hierarchical Attention Network
!8Yang et al. 2016. Hierarchical attention networks for document classification. (NAACL 2016)
Too Complicated! We simplify!
Page 9
!9
Time lineBefore Now
Dialogue StreamC A C A C A CA
W1 W2 Wn
……
……VTF−IDF
LSTM LSTM LSTM……Wtf−idf
btf−idf
a1
a2
an
Encoded Dialogue Representation External Feature ftf−idf
Softmax
FANBase
Watt
FANTF-IDF
FANfull
Model Framework• 😡 😵 😤
• ……
• ???
• !!!
• # of sentences / words
• # of words in two term dictionaries (complaint, 12315, customer right, ……)
• Dialog Sentiment
•
Page 10
Model Framework
!10
Time lineBefore Now
Dialogue StreamC A C A C A CA
W1 W2 Wn
……
……VTF−IDF
LSTM LSTM LSTM…… Watt
Wtf−idf
btf−idf
a1
a2
an
Encoded Dialogue Representation External Feature ftf−idf
Softmax
Page 11
!11
Evaluation Metrics
Recall@K =# of Detected Complainted Customers in Top K
# of Customer Complaints
Page 12
!12
Experimental Setup
• Comparison with Baselines
• Effect of Negative Samples
• Results over An Entire Week
• Online Deployment Results
Page 13
!13
Comparison with BaselinesLR-dict
LR-all
LightGBM
fastText
CNN
LSTM
FAN-base
FAN-tf-idf
FAN-full
0 6.5 13 19.5 26
Recall@5000
Page 14
!14
10M
5M
1M
0.1M
0 6.5 13 19.5 26
Recall@5000
Effect of Negative Samples
# of negative samples
Page 15
!15
0
10
20
30
40
June 17th June 18th June 19th June 20th June 21st June 22nd June 23rd
Average
Recall@5000
Results over An Entire Week
Page 16
!16
0
7.25
14.5
21.75
29
Oct 8th Oct 9th Oct 10th Oct 11st Oct 12nd Oct 13rd Oct 14th
Average
Recall@5000
Online Deployment Results
Page 17
!17
Lesson Learned
• Start with simple models.
• Don’t start over. Always reuse existing solutions.
• If NN cannot provide enough capacity, try manually-engineered features!
Page 18
!18
Q & A
Thank you!