An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie Vercoustre Stephen Wan PeterWilkins Ross Wilkinson Mingfang Wu
Apr 01, 2015
An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to
Organizational Websites
Cécile Paris
Anne-Marie Vercoustre Stephen Wan PeterWilkins
Ross Wilkinson Mingfang Wu
Outline
• Problem statement• System Description• Experiment• Discussion
Finding Information about a Corporation
• Often done through the web, searching and browsing
• Differences between results from such search-and-browse and printed brochures:– Printed brochures are coherent documents – and include
more than just information about a specific topic
– Printed brochures are typically tailored to an audience
Challenges in accessing an organization’s website
• Browse– know the organization’s structure
– know the web data structure
– otherwise -> users feel lost
• Search– It works for the “right” query or someone with the “right” mental
model how querying/searching works
– Otherwise, it may cause zero results or too many results
– Result lists is disorganized - each result is presented independently, although each result may be a piece of the overall picture
Our research question and approach
Can a tailored hypermedia system offering tailored and coherent information provide a good alternative to search-and-browse?
• Approach: Given a modest user model and a query:– Present information that not only matches the query but also is tailored
to the user.
– Present other information that is relevant to the context of the query (here information about an organization), like printed brochures do (e.g., introduction, contact details, etc.)
– Deliver the result as a concise and readable summary.
– Note that the summary may not cover all information a user needs, but is an informative guide that will lead the user to the needed information.
Our Tailor Hypermedia System: Percy
Dat
a In
terf
ace
User Model
Stereotype
Individual
Virtual Document Planner
DiscourseRules
Content Planner
DataSources
‘Documents’
VD
P D
ata
Inte
rfac
e
Inte
rfac
e S
peci
ficat
ion
Grammar(s)Lexicon(s)TemplatesStylesheets
SurfaceRealiser
PresentationRules
PresentationPlanner
Percy – user model
• Based on a requirements analysis• Contains:
– Stereotypical information• Current job position (e.g., scientist or CEO)
• Industry sector (e.g. science, engineering, finance)
– Individual information• Interests (as represented by a query term, e.g., language
technology)
• Preferred delivery channel (e.g web page, or hand-held device)
Percy – discourse planner
• Discourse rules or Schemas– They indicate which information to
present and how to organize it– They are used by the discourse
planner to build a discourse tree– Each rule extends a branch in the discourse tree– Coherence relations amongst
siblings indicate how the information fits together
Percy – presentation planner
• Discourse tree (which indicates content and organization) is extended with presentation markup
• Uses presentation plans
PPalm
C
PWeb
C
Percy – surface generator
• Traversal of final discourse tree to generate document
• Conversion from tree markup of layout to device dependent markup– Device dependent markup may
include use of further style sheetsSurface Realisation
XSLT templates
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
PWeb
C
PERCY – an example delivery to a scientistIntroduction to CSIRO
Short introduction to the divisions that have the projects
related to the query
Show the expanded query
Short description of the
matched project
PERCY – an example delivery to a scientist Link to the detailed description
Link to the onlinedemonstration
Publications
Contact person
PERCY – an example delivery to a CEO
Short introduction of how to do business with CSIRO
PERCY – an example delivery to a CEO
The project description and business relevance
Preliminary Evaluation
• Question Is this coherent and tailored delivery is useful interface to
organizational websites ?
Possible evaluation criteria
• User’s interactive experience– format
– content
• User’s attitudes towards the received message– intention
• The influence of the delivery message– The further interaction with the corporate
Experimental setting
• Test systems: PERCY vs. search-and-browse mechanism: a website search engine (Panoptic)
• Subjects: 20 university students• Search topics:
– T1: Human computer interaction
– T2: Mathematical modeling
– T3: Image analysis
– T4: Language technology
Evaluation instruments
• Post-system questionnaire (PSQ)– To get users’ opinion on the evaluation criteria
independently on each test system
• Exit questionnaire (EQ)– To get users’ opinion of the second system compared with
the first system
• All questions are evaluated on a 5-point Likert scale
Experiment design
• Latin square design to reduce the sequence effect of systems
Subjects System, topic
No. 1-5 Percy(t1,t2) PSQ Search (t3,t4) PSQ EQ
No. 6-10 Percy (t3,t4) PSQ Search (t1,t2) PSQ EQ
No. 11-15 Search(t1,t2) PSQ Percy (t3,t4) PSQ EQ
No. 16-20 Search (t3,t4) PSQ Percy (t1,t2) PSQ EQ
Post-system questionnaire – Measuring the content of the delivered information
• Q1: The system provides me accurate information
• Q2: The system provides sufficient information
• Q3: The information provided by the system meets my need.
• Q4: The system provides me comprehensive information
Q1 Q2 Q3 Q4
Search 3.50 3.60 3.35 3.70
Percy 3.65 3.30 3.65 3.55
Post-system questionnaire – Measuring the format of the delivered information
• Q5: The structure of the presented information is clear to me.
• Q6: I think the presented information is organized in a useful format.
• Q7: I think the presented information serves well as a useful online brochure.
Q5 Q6 Q7
Search 3.65 3.50 2.75
Percy 4.00 3.80 3.50
Post-system questionnaire – Understanding the intention of the user
• Q8: I would like to get more information on some specific projects presented.
• Q9: I would like to get more information because I have not got the needed information.
Q8 Q9
Search 3.95 3.15
Percy 3.90 2.90
Exit-questionnaire – Evaluating users’ preferences
Q1 Q2 Q3 Q4 Q5 Q6 Q7
Search 2.70 3.00 3.10 2.80 3.30 3.50 2.70
Percy 3.70 3.60 3.50 4.20 4.20 3.40 4.00
P < 0.01 0.03 0.04
focus o
n the s
earch
topic bett
er
make s
earch
task ea
sier
deliver
higher quali
ty info
attrac
t atte
ntion bett
er
provide bett
er ex
planati
on
provide more
compreh
ensiv
e info
prefere
nce
Summary of experimental results
• With Percy, subjects can better focus on their tasks.• With Percy, subjects understand better why a piece
of information is presented.• Subjects preferred Percy.
Discussion
• About tailored hypermedia(anything to say about the delivery method itself or limitations?)
– Delivery – who is in charge, user/author? or it is situation dependent? – How to apply AH to a dynamic resource (or even a different domain)
with minimum effort?– Is user’s expectation of AH system different from IR?
• About the evaluation– What to compare – Is it fair to compare the adaptive system with the
non-adaptive system? Or should we compare two adaptive systems? – Will the preference brings the enhancement of user performance? – (How to measure long term benefit?)