Page 1
Method for Atypical Opinion Extraction from Answers in Open-ended Questions(IEEE International Conference on
Computational Cybernetics ICCC 2004)
Ayako Hiramatsu Shingo TamuraOsaka Sangyo University Osaka University
Hiroaki Oiso Norihisa KomodaCodetoys K. K. Osaka University
Page 2
2/23
Abstract Introduction
Open-ended questions vs. closed-ended questions
Atypical opinions vs. typical opinions System
Aim 3 Methods: ratio, distance, phrases
Experiment Application experiment Evaluation experiment
Conclusion
Page 3
3/23
Introduction (1/5) Motivation:
Mobile game market has been expanding rapidly
Game providers need to attract more users and prolong the subscription period per user
Subscribers answer questionnaires when canceling their accounts
Closed-ended and open-ended questions Typical and atypical opinions
Page 4
4/23
Introduction (2/5) Target game: mobile quiz game
In Japanese Since 2002 3 carriers Questions are answered by choosing a
correct answer from 4 choices. If consumers unsubscribe, all
information is lost, and the questionnaire (closed-ended & open-ended questions) is given to be answered.
Page 5
5/23
Introduction (3/5)
Closed-ended questions: Users are asked to choose from a limited
number of pre-selected answers. Unable to acquire unexpected ideas Example:
Page 6
6/23
Introduction (4/5) Open-ended questions:
Consumers can freely write opinions. Not punctuated, ungrammatical, and
abbreviated Reveal dissatisfaction that cannot be captured in
the closed-ended questions. Few useful answers: most answers reflect
opinions already known by closed-ended questions
Time-consuming to read all of the texts Types: typical & atypical
Page 7
7/23
Introduction (5/5) Typical & atypical open-ended questions:
Page 8
8/23
System (1/10) Aim: a system that efficiently extracts
unexpectedly unique ideas by culling useless opinions from the data of open-ended question.
Outline:
(ChaSen)
words: noun, adj, verb
“packet “ + “fee” “packet fe
e”3 comparing
methodNext slide
atypical
typical
Page 9
9/23
System (2/10)
Typical word database:
Page 10
10/23
System (3/10) To extract atypical opinions Compare the keywords of each opinion
with the typical word database 3 methods:
Based on the ratio of typical word combinations in the sentences
Consider the word order and the distance of difference between the positions of words
Divide the opinion into phrases at each typical word combination
Page 11
11/23
System (4/10)
Method 1: ratio Remove opinions having neither
keyword nor a noun keyword Compare keywords with typical
elements (the combination in the typical word database)
Page 12
12/23
System (5/10)
Example:
Formula 1:
2+2×1≧4
2+2×1≦6
α=2
typical
Page 13
13/23
System (6/10)
Problems: Misrecognition method 2
Long sentence method 3
2+2×1≧4
Page 14
14/23
System (7/10)
Method 2: distance Keyword distance d : the position
difference of keywords Modify typical elements: keyword
distance is short, i.e. 2 keywords appearing near (d = 2)
Apply Formula 1
Page 15
15/23
System (8/10)
Example:
2+2×1≧4
0+2×0≦4
Page 16
16/23
System (9/10) Method 3: phrases Long sentences few atypical
elements should NOT be omitted sentences should be divided into phrases by delimiters
Delimiters: Punctuation mark pictograph (X) Typical elements (O)
Apply Formula 1 on phrases
Page 17
17/23
System (10/10)
Example:
Page 18
18/23
Application Experiment (1/2) Compare the three proposed methods Questionnaire data of users who
unsubscribed from a certain carrier for 7 months
Content provider classified 3263 opinions = 2993 typical & 270 atypical opinions
About 8000 kinds of word combinations were registered to the typical word database
Page 19
19/23
Application Experiment (2/2)
Result:
ANS 2993 270
phrases
distance
ratio
Page 20
20/23
Evaluation Experiment (1/2) Examine the best method: method 3 Questionnaire data of users who
unsubscribed from other carriers Content providers classified 1764
opinions = 1589 typical & 175 atypical opinions
The typical word database is the same as in the application experiment.
Page 21
21/23
Evaluation Experiment (2/2) Result:
The opinions with short sentences having 3 or 4 keywords low recall α=1 extract a huge number of atypical opinions low precision tradeoff
ANS 1589 175
LESSSatisfactor
y!
Page 22
22/23
Conclusion Described a support system for atypical
opinion extraction from answers in open-ended questions collected from consumers of mobile games when they unsubscribe
Proposed three methods of extraction of atypical opinions: ratio, distance, phrases
Differences of carriers also affect the accuracy of extraction.
Page 24
24/23
Delimiters insertion