Amélie Marian – Rutgers University 09/30/2013 Searching Web Forums Amélie Marian, Rutgers University Joint work with Gayatree Ganu
Feb 24, 2016
Amélie Marian – Rutgers University 09/30/2013
Searching Web Forums
Amélie Marian, Rutgers University
Joint work with Gayatree Ganu
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
2
Forum Popularity and Search
• Forums with most traffic [http://rankings.big-boards.com]- BMW
- 50K uniq visitors/day- 25M Posts- 0.6M Members
- Filipino Community- Subaru Impreza Owners- Rome Total War- …- Pakistan Cricket Fan Site- Prison Talk- Online Money making
Despite popularity, forums lack good search capabilities
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
3
Outline
Multi-Granularity Search
Challenges- Unstructured text- Background information omitted- Discussion digression
ContributionsReturn each results at varying focus levels, allowing more or less context. (CIKM 2013)
Egocentric Search
Challenges- Multiple interpersonal relations with
varying importance
ContributionsProposed a multidimensional user similarity measure.Use authorship for improving personalized and keyword search.
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
4
Hierarchical Model• Hierarchy over objects at three searchable levels
– pertinent sentences, larger posts, entire discussions or threads• Hierarchy
– captures strength of association, containment relationship
• Lower levels for smaller objects
• Edge represents containment
• Edge weight of 2 indicates that the text in child was repeated in the text of parent
Thread 1 Thread 2
Post 1 Post 2 Post 4Post 3
Sent 1 Sent 2 Sent 3 Sent 4 Sent 5 Sent 6
Dataset
Word 1 Word 2 Word 3 Word 4 Word 1
2
2
2
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
5
Alternate Scoring Functions
Example Textual Results. Query : hair loss
Top-4 Results
Post1: (A) Aromasin certainly caused my hair loss and the hair started falling 14 days after the chemo. However, I bought myself a rather fashionable scarf to hide the baldness. I wear it everyday, even at home. (B) Onc was shocked by my hair loss so I guess it is unusual on Aromasin. I had no other side effects from Aromasin, no hot flashes, no stomach aches or muscle pains, no headaches or nausea and none of the chemo brain.
Post2: (C) Probably everyone is sick of the hair loss questions, but I need help with this falling hair. I had my first cemotherapy on 16th September, so due in one week for the 2nd treatment. (D) Surely the hair loss can’t be starting this fast..or can it?. I was running my fingers at the nape of my neck and about five came out in my fingers. Would love to hear from anyone else have AC done (Doxorubicin and Cyclophosphamide) only as I am not due to have the 3rd drug (whatever that is - 12 weekly sessions) after the 4 sessions of AC. Doctor said that different people have different side effects, so I wanted to know what you all went through. (E) Have n’t noticed hair loss elsewhere, just the top hair and mainly at the back of my neck. (F) I thought the hair would start thining out between 2nd and 3rd treatment, not weeks after the 1st one. I have very curly long ringlets past my shoulders and am wondering if it would be better to just cut it short or completely shave it off. I am willing to try anything to make this stop, does anyone have a good recommendation for a shampoo, vitamins or supplements and (sadly) a good wig shop in downtown LA.
Post3: My suggestion is, don’t focus so much on organic. Things can be organic and very unhealthy. I believe it when I read that nothing here is truly organic. They’re allowed a certain percentage. I think 5% of the food can not be organic and it still can carry the organic label. What you want is nonprocessed, traditional foods. Food that comes from a farm or a farmer’s market. Small farmers are not organic just because it is too much trouble to get the certification. Their produce is probably better than most of the industrial organic stuff. (G) Sorry Jennifer, chemotherapy and treatment followed by hair loss is extremely depressing and you cannot prepare enough for falling hair, especially hair in clumps. (H) I am on femara and hair loss is non-stop, I had full head of thick hair.
tf*idfSent (E) (4.742)Sent (A) (4.711)Sent (C) (4.696)Sent (G) (4.689)
BM25Sent (D) (10.570)Sent (B) (10.458)Sent (H) (10.362)Sent (E) (10.175)
HScorePost2 (0.131)Sent (G) (0.093)Post1 (0.092)Sent (H) (0.089)
Score tf*idf (t,d) = (1+log(tft,d)) * log(N/dft) * 1/CharLength
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
6
Scoring Multi-Granularity ResultsGoal: Unified scoring for objects at multiple granularity levels
– largely varying sizes – with inherent containment relationship
Hierarchical Scoring Function (HScore)Score for node i with respect to search term t and having j children:
… if i is a non-leaf node= 1 … if i is a leaf node containing t= 0 … if i is a leaf node not containing t
ewij = edge weight between parent i and child jP(j) = number of parents of j C(i) = number of children of i
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
7
Effect of Size Weighting Parameter on HScore
• Parameter controls the intermixing of granularities
0 0.1 0.2 0.3 0.4 0.5 BM250
2
4
6
8
10
12
14
16
18
20
ThreadsPostsSentences
Size parameter
Num
ber o
f res
ults
in
top-
20 li
st
HScore
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
8
Multi-Granularity Result Generation
Sorted Ordering: Post3(2.5), Post1(2.1), Post2(2), Sent1(1.6), Sent2(1.5), Sent3(1.4), Sent4(1.3), Sent6(0.4), Sent5(0.1), Post4(0.1), Thread1(0.1), Thread2(0.1)
For result size k=4, optimizing for the sum of scores:• Overlap: {Post3, Post1, Post2, Sent1} Sum Score = 8.2 (minus 1.6?)• Greedy: {Post3, Post1, Post2, Sent6} Sum Score = 7.0• Best: {Post3, Post2, Sent1, Sent2} Sum Score = 7.633% sample queries had overlap amongst at least 3 of top-10 results
Thread 1 Thread 2
Post 1 Post 2 Post 4Post 3
Sent 1 Sent 2 Sent 3 Sent 4 Sent 5 Sent 6
0.1
2.1 2 2.5 0.1
0.1
0.1 0.41.6 1.5 1.4 1.3
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
9
Multi-Granularity Result Generation
Goal: Generating a non-overlapping result set maximizing “quality”
• Quality = Sum of scores of all results in the set• Maximal independent set problem (NP Hard)• Existing Algorithm: Lexicographic All Independent Sets (LAIS)
outputs maximal independent set with polynomial delay in specific order
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
10
Optimal Algorithm for k-set (OAKS)
• Fix node ordering by decreasing scores• Efficient OAKS Algorithm (typically k<<n):– Start with k-sized first independent set, i.e., greedy– Branch from nodes preceding kth node of the set, check if
maximal– Find new k-sized maximal sets, save in priority queue– Reject sets from priority queue where starting node occurs
after current best set’s kth node
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
11
OAKS
Sorted Ordering: Post3(2.5), Post1(2.1), Post2(2), Sent1(1.6), Sent2(1.5),
Sent3(1.4), Sent4(1.3), Sent6(0.4), Sent5(0.1), Post4(0.1), Thread1(0.1), Thread2(0.1)
For k=4, Greedy = {Post3, Post1, Post2, Sent6} SumScore=7.0 In the 1st iteration:{Post3, Post2, Sent1, Sent2} SumScore = 7.7{Post3 , Post1, Sent3, Sent4} SumScore = 7.3
Branches from nodes before Sent6, i.e. Sent1, Sent2, Sent3, Sent4Branch from Sent1, removing all adjacent to Sent1, {Post3, Post2, Sent1}Maximal on first 4 nodes? YES!then complete to size k and insert in queue- {Post3, Post2, Sent1, Sent2}
Thread 1 Thread 2
Post 1 Post 2 Post 4Post 3
Sent 1 Sent 2 Sent 3 Sent 4 Sent 5 Sent 6
0.1
2.1 2 2.5 0.1
0.1
0.1 0.41.6 1.5 1.4 1.3
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
12
Evaluating OAKS AlgorithmComparing OAKS RuntimeSmall overhead for practical k (=20)• Scoring time = 0.96 sec• OAKS Result set generation time = 0.09 sec
Word Frequency
Sets Evaluated Run Time (sec)
LAIS OAKS LAIS OAKS
20-30 57.59 8.12 0.78 0.12
30-40 102.07 5.06 7.88 0.01
40-50 158.80 5.88 26.94 0.01
50-60 410.18 6.30 82.20 0.02
60-70 716.40 5.26 77.61 0.01
70-80 896.59 8.30 143.33 0.04
Comparing LAIS and OAKS– 100 relatively infrequent queries
with corpus frequency in range 20-30, 30-40…
– OAKS is very efficient. Time required by OAKS depends on k
OAKS improves overGreedy SumScore in31% queries @top20
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
13
Dataset and Evaluation Setting• Data collected from breastcancer.org
– 31K threads, 301K posts, 1.8M unique sentences, 46K keywords• 18 Sample Queries
– e.g., broccoli, herceptin side effects, emotional meltdown, scarf or wig, shampoo recommendation …
• Experimental Search Strategies – top20 results- Mixed-Hierarchy : Optimal mixed granularity result.- Posts-Hierarchy : Hierarchical scoring of posts only.- Posts-tf*idf : Existing traditional search.- Mixed-BM25
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
14
Evaluating Perceived Relevance
Graded Relevance ScaleExactly relevant answer,Relevant but too broad,Relevant but too narrow,Partially relevant answer,Not Relevant
Crowd Sourced Relevance using Mechanical Turk
- Over 7 annotations- Quality control -Honey pot
questions- EM algorithm for consensus
Query = shampoo recommendation
= 0.1 = 0.2 = 0.3 = 0.4
Rank = 1 Rel Broad Rel Broad Rel Broad Partial
2 Rel Broad Rel Broad Rel Broad Partial
3 Rel Broad Rel Broad Rel Broad Partial
4 Rel Broad Rel Broad Exactly Rel Rel Broad
5 Rel Broad Rel Broad Exactly Rel Partial
6 Exactly Rel Exactly Rel Rel Narrow Rel Narrow
7 Rel Broad Exactly Rel Rel Narrow Not Rel
8 Rel Broad Rel Broad Not Rel Partial
9 Rel Broad Rel Narrow Rel Broad Partial
10 Exactly Rel Rel Narrow Partial Rel Narrow
11 Rel Broad Rel Broad Exactly Rel Not Rel
12 Rel Broad Rel Broad Exactly Rel Not Rel
13 Rel Broad Exactly Rel Partial Not Rel
14 Not Rel Exactly Rel Rel Narrow Partial
15 Not Rel Exactly Rel Not Rel Rel Broad
16 Not Rel Rel Broad Rel Narrow Not Rel
17 Exactly Rel Rel Broad Exactly Rel Not Rel
18 Exactly Rel Exactly Rel Partial Partial
19 Not Rel Rel Broad Rel Narrow Not Rel
20 Not Rel Exactly Rel Partial Not Rel
Mixed-Hierarchy
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
15
Evaluating Perceived Relevance
Mean Average PrecisionSearch System
MAP @
= 0.1
= 0.2
= 0.3
= 0.4
MixedHierarchy
10 0.98 0.98 0.90 0.70
20 0.97 0.95 0.85 0.66
Posts-Hierarchy
10 0.76 0.75 0.77 0.78
20 0.72 0.71 0.73 0.75
Posts- tf*idf
10 0.76 0.73 0.76 0.76
20 0.74 0.72 0.72 0.73
Mixed BM25
10 b=0.75 0.55
20 k=1.2 0.54
Clearly, Mixed-Houtperformspost only methods
Users perceive higherrelevance of mixedgranularity results
α=0.1 α=0.2 α=0.3 α=0.4 b=0.75 k1=1.2
0.00
5.00
10.00
15.00
20.00
25.00
30.00
35.00
Discounted Cumulative Gain @20
Mixed-hierarchy Posts-hierarchyPosts-tf*idf Mixed-BM25
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
16
EgoCentric Search• Previous technique did not take the authorship of posts into
account• Some forum participants are similar, sharing same topics of
interest or having the same needs, not necessarily at the same time– Rank similar author’s posts higher for personalized search
• Some forum participants are experts, prolific and knowledgeable– Expert opinions carry more weight in keyword search
• Author score to enhance personalized & keyword search
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
17
Author Score• Forum participants have several reasons to be linked
• Build a multidimensional heterogeneous graph over authors incorporating many relations
• But, users assign different importance to different relations
auth 1Topic 1
auth 2
auth n
Topic 2
Topic t
query 1
query 2
query n
W(a,t) W(q,t) author 1
author 2
author n
author 3
W(a1,a2)
User Profiles:
- Location- Age- Cancer stage- Treatment- …
-Co-participation-Explicit References
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
18
ContributionsCritical problem for leveraging authorship for search:
Incorporating multiple user relations with varying importance learned egocentrically from user behavior
Outline:• Author score computation using multidimensional graph• Personalized predictions of user interactions: authors most
likely to provide answers• Re-ranking results of keyword search using author expertise
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
19
Multi-Dimensional Random Walks (MRW)
• Random Walks (RW) for finding most influential users– Pt+1 = M × Pt … till convergence– M = α(A + D) + (1 − α)E … relation matrix A, D for dangling
nodes, uniform matrix E, α usually set to 0.85
• Rooted RW for node similarity – Teleport back to root node with probability (1-α)– Computes similarity of all nodes w.r.t root node
• Multidimensional RW– Heterogeneous Networks:– Transition matrix computed as A = 1 * A1 + 2 * A2 + ... + n * An
where i i = 1 and all i >= 0– Egocentric weights -
For root node r : i (r) = j ewAi (r, m)/ Ak j ewAk (r, j)… m Ai and j
Ak
a
b
c
2
3
A =a b c
a 0 0 0
b 2 0 0
c 0 3 0
D =a b c
a 0 0 0.33
b 0 0 0.33
c 0 0 0.33
E =
a b c
a .33 .33 .33
b .33 .33 .33
c .33 .33 .33
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
20
Personalized Answer Search• Link prediction by leveraging user similarities:
– Given participant behavior, find similar users to the user asking question– Predict who will respond to this question
• Learn similarities from first 90% training threads• Relations used:
– Topics covered in text, Co-participation in threads, Signature profiles, Proximity of posts
• MRW similarity compared with baselines:– Single relations– PathSim:
• Existing approach for heterogeneous networks• Predefined paths of fixed length• No dynamic choice of path
Link prediction enablessuggesting which threadsor which users to follow
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
21
Predicting User Interactions
10 20 30 40 50 60 70 80 90 1000
0.1
0.2
0.3
0.4
0.5
Top-K similar participants
MAP
MAP for link prediction
Multidimensional RWhas best predictionperformance
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
22
Predicting User Interactions• Leverage content of the initial post to find users who are
experts on the question– TopicScore computed as cosine similarity between author’s history and
initial post– UserScore = β * MRWScore + (1- β) * TopicScore
Neighbors β = 0 β = 0.1 β = 0.2 β = 1
Top 5 0.52 0.64 (8%) 0.61 (4%) 0.59
Top 10 0.31 0.50 (8%) 0.49 (5%) 0.46
Top 15 0.24 0.43 (8%) 0.42 (6%) 0.40
Top 20 0.20 0.39 (6%) 0.39 (7%) 0.37
Purely MRWPurely topical
expertise
% Improvement over purely MRW
MAP
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
23
0 0.1 0.2 0.3 0.4 0.5
0.600000000000001
0.700000000000001 0.8 0.9 10.72
0.74
0.76
0.78
0.80
IR Score λ=0.1IR Score λ=0.2
Tradeoff Parameter ω
MAP
@ 1
0Enhanced Keyword Search• Non-rooted RW to find most influential expert users• Re-rank top-k results of IR scoring using author scores• Final score of post = ω*IR_score λ + (1- ω)*Authority_score
– Posts only, tf*idf scoring with size parameter
Re-ranking searchresults with authorscore yields higherMAP relevance
4% improvement
5%
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
24
Patient Emotion and stRucture Search USer tool(PERSEUS) - Conclusions
• Designed hierarchical model and score that allows generating search results at several granularities of web forum objects.
• Proposed OAKS algorithm for best non-overlapping result. • Conducted extensive user studies, show that mixed collection of
granularities yields better relevance than post-only results. • Combined multiple relations linking users for computing similarities• Enhanced search results using multidimensional author similarity• Future Directions:
– Multi-granular search on web pages, blogs, emails. Dynamic focus level selection.– Search in and out of context over dialogue, interviews, Q&A.– Optimal result set selection for targeted advertising, result diversification– Time sensitive recommendations – Changing friendships, progressive search
needs.
Amélie Marian – Rutgers University 09/30/2013
Thank you!
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
26
Why Random Walks?Rooted RW Examples
r t(a)
r a(b) t
r a(c) t
b c
r a(d) t
b
0.4
0.16
0.26
c u0.16
Multi-dimensional Rooted RW Example
0.6
r1
score (b w.r.t r1) = 0.072score (c w.r.t r1) = 0.096
score (b w.r.t r2) = 0.097score (c w.r.t r2) = 0.066
r22
b1
4
A1
r1
r24
c1
2
A2
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
27
LAIS
Sorted Ordering: Post3(2.5), Post1(2.1), Post2(2), Sent1(1.6), Sent2(1.5), Sent3(1.4), Sent4(1.3), Sent6(0.4), Sent5(0.1), Post4(0.1), Thread1(0.1), Thread2(0.1)
Greedy = {Post3, Post1, Post2, Sent6}. In the 1st iteration:{Post3, Post2, Sent1, Sent2, Sent6}{Post3, Post1, Sent3, Sent4, Sent6}{Post1, Post2, Sent6, Sent5}{Post3, Post1, Post2, Post4){Post3, Sent6, Thread1}{Post1, Post2, Thread2}
Thread 1 Thread 2
Post 1 Post 2 Post 4Post 3
Sent 1 Sent 2 Sent 3 Sent 4 Sent 5 Sent 6
0.1
2.1 2 2.5 0.1
0.1
0.1 0.41.6 1.5 1.4 1.3
2 2
Amélie Marian – Rutgers University 09/30/2013
Gayatree Ganu
28
Current Search Functionality breastcancer.org
Filtering criteriakeyword search,member search
Ranking basedon date
Posts onlyresults