CLT 2009, Heerlen, 03.03.09 The Effects of an Exploratory Web Search Interface on Performance and Cognitive Load Yvonne Kammerer, Peter Pirolli, Rowan Nairn, Peter Gerjets [email protected]
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
CLT 2009, Heerlen, 03.03.09
The Effects of an Exploratory Web Search Interface
on Performance and Cognitive Load
Yvonne Kammerer,Peter Pirolli,
Rowan Nairn,Peter Gerjets
CLT 2009, Heerlen, 03.03.09
The World Wide Web – A Huge Information Source
Ill-structered,multi-faceted
problemsOpen-endedinformation
goals
CLT 2009, Heerlen, 03.03.09
LookupInvestigate Learn
Fact retrievalKnown item searchVerificationQuestion answering
…
Knowledge acquisitionComprehensionInterpretationComparison
…
EvaluationAnalysisDiscoverySynthesis
…
Explore
Exploratory Search
Marchionini (2006)
CLT 2009, Heerlen, 03.03.09
Exploratory Search Process – Continuous Process
Submission of a tentative query
Decision making aboutwhich paths to follow
Exploration of returned hits
Selection of information
Query refinements
Learningtaking placeduring the process
White, Marchionini, & Muresan (2006)
multiple iterations
CLT 2009, Heerlen, 03.03.09
Exploratory Search Process
Difficulties for searchers:
• Search goal is vague
Uncertainty about what to search for and how to search
• Huge amount of information presented
Uncertainty about which information is relevant
CLT 2009, Heerlen, 03.03.09
Exploratory Search Interfaces
Aims:• Support searchers in their exploratory search
• Help searchers explore complex information spaces
• Help searchers learn about new topics: go beyond fact-finding
• Improve searchers’ queries
• Support searchers’ in navigation and decision-making
CLT 2009, Heerlen, 03.03.09
Features of Exploratory Search Interface
other keywords relatedto this web page
arrowsto give
relevance feedback
relatedkeywordslist
Providing related keywords
User-providedrelevance feedback
CLT 2009, Heerlen, 03.03.09
Baseline Version
CLT 2009, Heerlen, 03.03.09
Hypotheses
1. Performance:Exploratory search interface > Baseline interfaceExploratory search interface supports searchers’ performance (search outcome and informal learning)
• Cognitive load: Exploratory search interface > Baseline interfaceInteraction with exploratory search interface increases searchers’ cognitive load.
germane cognitive load?
CLT 2009, Heerlen, 03.03.09
Experimental Design
Factor 1: Search Interface
Factor 2: Task Domain
Future Architecture CLPerformance
CLPerformance
Global Warming
Web Mashups
Exploratory interface (n=15)(related keywords,
relevance feedback)
Baseline interface (n=15)(no exploratory search
features)
CLPerformance
CLPerformance
CLPerformance
CLPerformance
+interactivefeature
use
Participants:22 male, 8 femalePARC members and Stanford studentsaverage age: 31.9 years (range: 21-54)
CLT 2009, Heerlen, 03.03.09
Performance Measures: Tasks
1. Page Collection tasks: Collecting as many relevant pages as possible to specific pre-defined queries
2. Summarization tasks: Writing general summaries about the domain of search
3. Keywords Formulation tasks: Formulating keywords for search
CLT 2009, Heerlen, 03.03.09
Cognitive Load Measure: NASA Task Load Index
5 sub scales from 0=very low to 100=very high :• mental demand • time pressure • perceived success• mental effort • stress level
Aggregation of all 5 sub scales (with perceived success inverted):• Overall Cognitive Load measure
0=very low cognitive load to 100=very high cognitive load
Hart & Staveland, 1988
Cronbach‘s alpha= .83
CLT 2009, Heerlen, 03.03.09
Domain 1:
Page Collection Task (6 min)Summarization Task (12 min)
Keywords Task (2 min)
NASA-TLX
Instructions and Training Tasks
Pre-Questionnaire(demographics, prior knowledge)
Domain 2: …
Domain 3: …
Duration: approx. 1.5 hUsability ratings
Differences in prior knowledge about Web mashups
covariate
Experimental Procedure
CLT 2009, Heerlen, 03.03.09
Results: Performance Data – Summarization Tasks
Summarization task performance
Interface:F (1, 26) = 8.91, p = .01, η2 = .26
Mashups Knowledge:ns.
Interface x Mashups Knowledge:F (1, 26) = 3.68, p = .07, η2 = .12
0
1
2
3
4
5
futurearchitecture globalwarming mashups
num
ber o
f sum
mar
y as
pect
s
baseline exploratory
p=.11
*
**
CLT 2009, Heerlen, 03.03.09
Relationship: Mashups Summaries and Mashups Knowledge
r=.46(*)
r=-.18
CLT 2009, Heerlen, 03.03.09
Results: Performance Data – Keywords Tasks
Keywords task performance
Interface:ns.
WM Knowledge:ns.
Interface x Mashups Knowledge:F (1, 26) = 5.06, p = .03, η2 = .16
0
2
4
6
8
10
12
futurearchitecture globalwarming mashups
num
ber o
f key
wor
ds
baseline exploratory
CLT 2009, Heerlen, 03.03.09
Relationship: Mashups Keywords and Mashups Knowledge
r=.05
r=.74**
CLT 2009, Heerlen, 03.03.09
Results: Cognitive Load
Cognitive load
Interface:F (1, 26) =6.23, p = .02, η2 = .19
WM Knowledge:ns.
Interface x Mashups Knowledge:ns.
0
20
40
60
80
100
futurearchitecture globalwarming mashups
cogn
itive
load
baseline exploratory
* *p=.13
CLT 2009, Heerlen, 03.03.09
Results: Interactive Feature Use
Average interactive feature use per task:
• M = 3.81 times (SD = 2.24)• Page Collection tasks: M = 5.54 times (SD = 3.03)• Summarization tasks: M = 2.07 times (SD = 1.71)
Positive correlation between feature use and cognitive load• r = .59 (p = .02)
The more the exploratory features were used,the higher was the cognitive load.
CLT 2009, Heerlen, 03.03.09
Relationship between Cognitive load, Performance, and Interactive Feature Use for Summarization Tasks
InteractiveFeature use
Cognitiveload
Summaryperformance
r=.38(p=.16)
r=-.32(p=.24)
For Exploratory interface users (n=15)
r=-.37(p=.17)
rpart=-.28(p=.33)
CLT 2009, Heerlen, 03.03.09
Summary and Conclusion
Performance:Exploratory search interface > Baseline interfaceExploratory search system supports searchers’ performance
• Particular support for domain novices
Cognitive load: Exploratory search interface > Baseline interface
• Interaction with exploratory search interface increases searchers’cognitive load
However, interpretation as germane cognitive load not warranted
CLT 2009, Heerlen, 03.03.09
Thanks for your attention!
http://mrtaggy.com
CLT 2009, Heerlen, 03.03.09
Backup
CLT 2009, Heerlen, 03.03.09
Exploratory Search Systems
Exploratory Search Features:Tools to support navigation and decision-making
• System provides options, e.g. keywords, to choose from when searching
• Relevance feedback for query refinements(from the user to the system)
CLT 2009, Heerlen, 03.03.09
Benefits and Costs of Exploratory Search Systems
Presentation of additional information and support of rich interactionimproves the relevance of the documents found
High complexity of interface increases cognitive load and search time
Bruza, McArthur, & Dennis (2000)
CLT 2009, Heerlen, 03.03.09
Summarization Tasks
Summarization tasks: writing more general summaries about the domain of search
Domain Summarization task
Future Architecture
Describe three different topics what "Future Architecture" could be about
Global Warming
Arguments or evidence in favor and against human-caused Global Warming
Web Mashups Benefits of the use of Web Mashups
CLT 2009, Heerlen, 03.03.09
Subscales 2-5 of NASA TLX
CLT 2009, Heerlen, 03.03.09
Results: Performance Data – Page Collection tasks
0
1
2
3
4
5
6
7
8
9
10
futurearchitecture globalwarming mashups
num
ber o
f pag
es c
olle
cted
baseline exploratory
Pages Collection task performance
Interface:ns.
Mashups Knowledge:ns.
Interface x Mashups Knowledge:ns.
CLT 2009, Heerlen, 03.03.09
Results: Time on Task for Summary Tasks
0
2
4
6
8
10
12
14
futurearchitecture globalwarming mashups
time
take
n fo
r sum
mar
ies
(in s
ec)
baseline exploratory
**
**p=.18
Time on task
Interface:F (1, 26) = 13.04, p = .001, η2 = .33
Mashups Knowledge:ns.
Interface x Mashups Knowledge:ns.
CLT 2009, Heerlen, 03.03.09
Results: Performance Data – Summarization tasks
Summarization task performance
With time on task as covariate
Interface:F (1, 26) = 6.13, p = .02, η2 = .20
Summary time:ns.
Interface x Summary time:ns.
Mashups Knowledge:ns.
Interface x Mashups Knowledge:F (1, 26) = 3.68, p = .07, η2 = .12
0
1
2
3
4
5
futurearchitecture globalwarming mashups
num
ber o
f sum
mar
y as
pect
s
baseline exploratory
p=.12
*
(*)
CLT 2009, Heerlen, 03.03.09
Results: Cognitive load (MANCOVA with 5 subscales)
Interface:(5, 22) = 2.24, p = .09, η2 = .34
Univariate main effects of Interface:• Mental demand: p = .21
• Time pressure: p = .01
• Perceived success: p = .08
• Effort: p = .11
• Stress level: p = .06
CLT 2009, Heerlen, 03.03.09
Relationship between Cognitive load, Performance, and Interactive Feature Use for Page Collection Tasks
InteractiveFeature use
Cognitiveload
Page Coll.performance
r=.61(p=.02)
r=.13(p=.65)
For Exploratory interface users (n=15)
r=-.07(p=.81)
rpart=-.19(p=.52)
Related Documents
