Understanding the Impact of the Role Factor in CIR Lynda Tamine and Laure Soulier 1
Apr 15, 2017
Understanding the Impact of the Role Factor in CIRLynda Tamine and Laure Soulier
1
Outline
• Collaborative search in context
• Study Objectives
• Study Design
• Results
• Design Implications
• Conclusion
Collaborative Search | Objectives | Design | Results | Implications | Conclusion 2
Collaborative search in context
333
Collaborative Search
Complex, exploratory or fact-finding tasksBibliographic, medical, e-Discovery, academic search…
What?
Requirement or setup need Shared interestsInsufficient knowledge Division of labor
Why?
Groups vs. Communities
Who?
Synchronous vs. Asynchronous
When?
Colocated vs. Remote
Where?
Crowd-sourcing User mediationImplicit vs. Explicit intent System mediation
How?
Collaborative Search | Objectives | Design | Results | Implications | Conclusion 4444
Collaborative Search Models
Collaborative Search | Objectives | Design | Results | Implications | Conclusion
Users' activities in social search [Evans and Chi, 2010]
AFTE
RBE
FORE
DU
RIN
G
● Refining the information need● Structuring the task guidelines
● Sense-making process : Querying Reading Extracting Exchanging...
Organizing/Distributing search outcomesAssessing collective relevance
Collaborative search practices [Morris, 2008]
Communication channels [Gonzalez-Ibanez et al., 2013]
Other behavioral models [Hyldegärd, 2009 ; Karunakaran et al., 2013 ; Shah, 2012]
5
Collaborative Search Paradigms
Collaborative Search | Objectives | Design | Results | Implications | Conclusion
Sharing of knowledge[Foley and Smeaton, 2009]
Division of labor[Kelly and Payne, 2013]
Awareness[Dourish and Bellotti, 1992]
Role-based division of labor
Document-based division of labor
Communication and shared workspace
Ranking-based on relevance judgments
Collaborator's actions
Spatio-temporal context
Shared information
need Doc2 Yes, good
Doc2 Yes, good
6
Role-based Collaborative Search
Collaborative Search | Objectives | Design | Results | Implications | Conclusion
DescriptionNaturalness Collaboration mediation Flexibility level
Exchange Interfaces Algorithmic approaches Role Algorithmic
User-drivencollaboration
Fixed roles [Imazu et al., 2011] = ++ + -- - - Freely-negociated roles [Morris et al., 2008] ++ ++ + -- + -
System-mediated collaboration
Fixed roles leveraged by algorithmic mediation [Pickens et al., 2008, Shah et al., 2010]
- = + ++ - +
User-driven system-mediated collaboration
Fixed roles : - freely negotiated- dynamically mined- leveraged by algorithmic mediation[Soulier et al., 2014]
+ = + ++ + +
Role taxonomy [Golovchinsky et al., 2009]
● Prospector/Miner● Gatherer/Surveyor● Domain A expert/Domain B expert● Search expert/Search novice
→ Analyzing the impact of the role factor in collaborative search
7
Study Objectives
8
Research Questions and Hypothesis
Understanding differences in users' behaviors in role-oriented and non-role-oriented collaborative search sessions:
● RQ1 : How do user search behaviors differ ?
● RQ2 : If any, does the coordination between collaborators differ ?
● RQ3 : Do users with assigned roles respect the guidelines? If no, why ?
● RQ4 : Which effect on the search effectiveness?
Research Hypothesis:• Behaviors of collaborators might be complementary w.r.t. division of labor policies
[Spence et al., 1995]
• Differences in collaborators' behaviors represent complementary signals [Soulier et al., 2014]
Collaborative Search | Objectives | Design | Results | Implications | Conclusion 9
Study Design
10
Participants
75 pairs of users (undergrad and Ph.D. students between 18 and 30)● Already know each other● Already worked on collaborative project together● Experience in browsing the web and using search engines
Compensation● Material compensation ($20)● Additional gift ($50) for three most effective groups
Collaborative Search | Objectives | Design | Results | Implications | Conclusion 11
System
Open-source Coagmento collaborative search system
Collaborative Search | Objectives | Design | Results | Implications | Conclusion 12
Tasks
Exploratory search taskThe mayor of your countryside village must choose between building a huge industrial complex or developing a nature reserve for animal conservation. As forest preservationists, you must raise awareness about the possibility of wildlife extinction surrounding such an industrial complex. Yet, before warning all citizens, including the mayor, you must do extensive research and collect all the facts about the matter. Your objective is to create a claim report together, outlining all the possible outcomes for wildlife should the industrial complex be built. Your focus is on wildlife extinction. You must investigate the animal species involved, the efforts done by other countries and the association worldwide to protect them and the reasons we, as humans, must protect our environment in order to survive. You must identify all relevant documents, facts, and pieces of information by using bookmarks, annotations, or saving snippets. If one document discusses several pieces of useful information, you must save each piece separately using snippets. Please assume that this research task is preliminary to your writing, enabling you to provide all relevant information to support your claims in your report.
3 settings associated to 3 TREC topics:● W/oRole (any role guidelines) – topic 408i (tropical storm)
● PM (prospector/miner) – topic 392i (robotics)
● GS (gatherer/surveyor) – topic 347i (wildfire extinction)
Collaborative Search | Objectives | Design | Results | Implications | Conclusion 13
User Study Workflow and Data
Step 1 : Sign-up questionnaire ● Users' characteristics● Users' habits in search browsing (search level and role assignment)● Users' habits in collaborative work
Step 2 : The training step ● System tutorial● Short time-period to test the system
Step 3 : The 30-minute search task ● Real task in which search logs where collected
Step 4 : Post-task questionnaire ● Opinion about CIS● Difficulty level of the task
Collaborative Search | Objectives | Design | Results | Implications | Conclusion
npq Average number of visited pages by query
dt Average time spent between two visited pages
nf Average number of relevance feedback (snippets, annotations & bookmarks)
qn Average number of submitted queries
ql Average number of query tokens
qo Average number of shared tokens among successive queries
nbm Average number of exchanged messages
14
Results
15
Users’ Behavioral Differences
Collaborative Search | Objectives | Design | Results | Implications | Conclusion
Modeling collaborators’ search behaviors [Soulier et al., 2014]
Bookmarked document
…
…
Annotated document
Annotated document
q2
q1
q3
q4
Snipped document
Shared information
need
● Avoiding noisy search actions
● Behaviors change
Search feature-based representation Temporal-based representation
Su1(t) (wu1, f1
(t) ,...,wu1, fn
(t) )
Su2(t) (wu2 , f1
(t) ,...,wu2 , fn
(t) ) Su1(t)
wu1, f1
(1) ... wu1, fn
(1)
... ... ...
wu1, f1
(t) ... wu1, fn
(t)
Su2(t)
wu2 , f1
(1) ... wu2 , fn
(1)
... ... ...
wu2 , f1
(t) ... wu2 , fn
(t)
16
Users’ Behavioral Differences
Collaborative Search | Objectives | Design | Results | Implications | Conclusion
Users with assigned roles significantly behave differently
Differences between W/Role and W/oRole settings● W/oRole spent longer time on pages and submit more queries
● W/Role annotated/bookmarked/snipped more
Query overlap explained by the role guidelines
Communication seems to be similarly distributed among scenarios
Objective: Analyzing the impact of the role factor on users' behavior
Slow and exhaustive evaluation style common among less (skill-based) experienced users
17
Users’ Behavioral Differences
Collaborative Search | Objectives | Design | Results | Implications | Conclusion
Mostly coordinative and content messages
● Coordination more important for W/oRole
● Content more important for W/Role●
Users in PM scenario need to coordinate more than those in GS one
Objective: Analyzing the impact of the role factor on communication channels
- Bridging the gap between individual and collaborative perception of the task- Role negotiation for W/oRole
Some roles might lead to ambiguity
18
Users’ Division of Labor Strategies
Collaborative Search | Objectives | Design | Results | Implications | Conclusion
f1
f2
f3
f4
f1
f2
Δf3
f4
*Δf1
Δf2
*Δf3
*Δf4
Difference significance test (Kolmogorov-Smirnov)
Δf1
f3
Δf4
Δf3
Δf1
Δf4
1 0.3 -0.5
0.3 1 -0.8
-0.5 -0.8 1
✔ Step 1: Identifying search behavior differences ✔ Step 2: Characterizing users’ roles
Correlations on search behavior differences for:- Highlighting search skill oppositions- Identifying in which each collaborator is the most
effective
Objective: Analyzing the impact of the role factor on users' search strategies
Investigating and contrasting complementarities of intra-group users' behavior [Soulier et al., 2014]
● Differences between users' behaviors
● Negative correlations between users' differences
19
Users' division of labor strategies
Collaborative Search | Objectives | Design | Results | Implications | Conclusion
Prospector-Miner Gatherer-Surveyor Without Role
Objective: Analyzing the impact of the role factor on users' search strategies
Higher number of significant correlated pairwise features in W/oRole
Exhibiting significant differences and reach convergence● PM: 2 of the 4 pairwise features varied between negative/null correlations
● GS: after 20-25 min
● W/oRole : after 15 min
Roles seem weakened the interdependence between users:- no coordination stability/role drift for PM- encounter cohesion for GS
Higer flexibility for W/oRole
20
Users' division of labor strategies
Collaborative Search | Objectives | Design | Results | Implications | Conclusion
Task generally perceived as difficult
A trend for easy in the W/oRole scenario, not assessed by the Chi-Square test
Task difficulty mainly related to page content, organization, and role ● Users in W/oRole feel that organization is more difficult
● Role guidelines might be difficult to follow
Objective: Analyzing the impact of the role factor on search task perception
21
Search Effectiveness
Collaborative Search | Objectives | Design | Results | Implications | Conclusion
Formula [Shah and Gonzalez-Ibanez, 2011]
Precision
Recall
F-measure
Objective: Analyzing the impact of the role factor on the search task effectiveness
P(g)=RelevantCoverage(g)
Coverage (g)
R(g)=RelevantCoverage (g)
GroundTruth
F (g)=2⋅P(g)⋅R(g)
P(g)⋅R (g)
Effectiveness higher for W/oRole only for the precision measure● More likely to spend more time on pages
● Less involved in making assessments
● Less communicative about the document content
But, users in both settings able to identify relevant pages
More successful in discarding irrelevant pages
22
Design Implications
23
Mining Latent Evolving Roles
Findings● Users in W/Role settings found difficult to follow their role guidelines● Users in W/oRole setting were able to coordinated even though coordination relied
on intensive coordination
Opened perspectives● Enhanced the user-driven system-mediated approach [Soulier et al., 2014] by :
Mining users' unalbelled and latent roles in a « just in time fashion » Reinjecting these unlabelled and latent roles in CIR algorithms
Collaborative Search | Objectives | Design | Results | Implications | Conclusion 24
Providing Task-based Role Template
Collaborative Search | Objectives | Design | Results | Implications | Conclusion
Findings● Task coordination messages are important for both settings
Opened perspectives● Designing content-based role template adapted to the search task type
Enhancing functional roles (PM, GS, ...) Reducing costs of organizing division of labor search strategies
Detecting collaborators' search intent and knowledge differences Refining users' browsing behaviors by distributing information nuggets among
collaborators Guiding collaborators in a « step-by-step »-oriented search according to their knowledge
25
Mining Latent Evolving Roles
Findings● Users in W/Role settings found difficult to follow their role guidelines● Users in W/oRole setting were able to coordinated even though coordination relied
on intensive coordination
Opened perspectives● Enhanced the user-driven system-mediated approach [Soulier et al., 2014] by :
Mining users' unalbelled and latent roles in a « just in time fashion » Reinjecting these unlabelled and latent roles in CIR algorithms
Collaborative Search | Objectives | Design | Results | Implications | Conclusion 26
Enhancing Role-based Awareness
Collaborative Search | Objectives | Design | Results | Implications | Conclusion
Findings● Users in W/Role settings found difficult to follow their role guidelines● Roles seems to reduce coordination-based communication costs
+ Fixed roles might be required for some tasks (e-Discovery, medical, ...)
Implications● Improving awareness w.r.t. role drift
Identifying indicators associated to each pair of roles Designing interfaces detecting rôle drift and monitoring users' support in order to better
follow role specifications
27
Conclusion
28
Conclusion
User-study of collaborative and exploratory search tasks constrained (or not) by role guidelines
– Users' search behaviors
– Users' division of labor strategies
– Search effectiveness
Quantitative and qualitative analysis highlighting : – Users without prior roles adopt a slow and exhaustive search
– Users without prio roles were more likely to broaden their complementarities and structure their role relatively early
– Roles seems to limit the precision of the search results
Discussion about promising perspective in CIR/CIS
Limitations of the work :
– Collaboration in dyads
– Exploratory search tasks
Collaborative Search | Objectives | Design | Results | Implications | Conclusion 29
30
See you on Friday!
Leif Azzopardi Jeremy Pickens Tetsuya Sakai Laure Soulier Lynda Tamine
Thank you for your attention
Do you have a question?
31