An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to
Organizational Websites
Cécile Paris
Anne-Marie Vercoustre Stephen Wan PeterWilkins
Ross Wilkinson Mingfang Wu
Outline
• Problem statement• System Description• Experiment• Discussion
Finding Information about a Corporation
• Often done through the web, searching and browsing
• Differences between results from such search-and-browse and printed brochures:– Printed brochures are coherent documents – and include
more than just information about a specific topic
– Printed brochures are typically tailored to an audience
Challenges in accessing an organization’s website
• Browse– know the organization’s structure
– know the web data structure
– otherwise -> users feel lost
• Search– It works for the “right” query or someone with the “right” mental
model how querying/searching works
– Otherwise, it may cause zero results or too many results
– Result lists is disorganized - each result is presented independently, although each result may be a piece of the overall picture
Our research question and approach
Can a tailored hypermedia system offering tailored and coherent information provide a good alternative to search-and-browse?
• Approach: Given a modest user model and a query:– Present information that not only matches the query but also is tailored
to the user.
– Present other information that is relevant to the context of the query (here information about an organization), like printed brochures do (e.g., introduction, contact details, etc.)
– Deliver the result as a concise and readable summary.
– Note that the summary may not cover all information a user needs, but is an informative guide that will lead the user to the needed information.
Our Tailor Hypermedia System: Percy
Dat
a In
terf
ace
User Model
Stereotype
Individual
Virtual Document Planner
DiscourseRules
Content Planner
DataSources
‘Documents’
VD
P D
ata
Inte
rfac
e
Inte
rfac
e S
peci
ficat
ion
Grammar(s)Lexicon(s)TemplatesStylesheets
SurfaceRealiser
PresentationRules
PresentationPlanner
Percy – user model
• Based on a requirements analysis• Contains:
– Stereotypical information• Current job position (e.g., scientist or CEO)
• Industry sector (e.g. science, engineering, finance)
– Individual information• Interests (as represented by a query term, e.g., language
technology)
• Preferred delivery channel (e.g web page, or hand-held device)
Percy – discourse planner
• Discourse rules or Schemas– They indicate which information to
present and how to organize it– They are used by the discourse
planner to build a discourse tree– Each rule extends a branch in the discourse tree– Coherence relations amongst
siblings indicate how the information fits together
Percy – presentation planner
• Discourse tree (which indicates content and organization) is extended with presentation markup
• Uses presentation plans
PPalm
C
PWeb
C
Percy – surface generator
• Traversal of final discourse tree to generate document
• Conversion from tree markup of layout to device dependent markup– Device dependent markup may
include use of further style sheetsSurface Realisation
XSLT templates
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
PWeb
C
PERCY – an example delivery to a scientistIntroduction to CSIRO
Short introduction to the divisions that have the projects
related to the query
Show the expanded query
Short description of the
matched project
PERCY – an example delivery to a scientist Link to the detailed description
Link to the onlinedemonstration
Publications
Contact person
PERCY – an example delivery to a CEO
Short introduction of how to do business with CSIRO
PERCY – an example delivery to a CEO
The project description and business relevance
Preliminary Evaluation
• Question Is this coherent and tailored delivery is useful interface to
organizational websites ?
Possible evaluation criteria
• User’s interactive experience– format
– content
• User’s attitudes towards the received message– intention
• The influence of the delivery message– The further interaction with the corporate
Experimental setting
• Test systems: PERCY vs. search-and-browse mechanism: a website search engine (Panoptic)
• Subjects: 20 university students• Search topics:
– T1: Human computer interaction
– T2: Mathematical modeling
– T3: Image analysis
– T4: Language technology
Evaluation instruments
• Post-system questionnaire (PSQ)– To get users’ opinion on the evaluation criteria
independently on each test system
• Exit questionnaire (EQ)– To get users’ opinion of the second system compared with
the first system
• All questions are evaluated on a 5-point Likert scale
Experiment design
• Latin square design to reduce the sequence effect of systems
Subjects System, topic
No. 1-5 Percy(t1,t2) PSQ Search (t3,t4) PSQ EQ
No. 6-10 Percy (t3,t4) PSQ Search (t1,t2) PSQ EQ
No. 11-15 Search(t1,t2) PSQ Percy (t3,t4) PSQ EQ
No. 16-20 Search (t3,t4) PSQ Percy (t1,t2) PSQ EQ
Post-system questionnaire – Measuring the content of the delivered information
• Q1: The system provides me accurate information
• Q2: The system provides sufficient information
• Q3: The information provided by the system meets my need.
• Q4: The system provides me comprehensive information
Q1 Q2 Q3 Q4
Search 3.50 3.60 3.35 3.70
Percy 3.65 3.30 3.65 3.55
Post-system questionnaire – Measuring the format of the delivered information
• Q5: The structure of the presented information is clear to me.
• Q6: I think the presented information is organized in a useful format.
• Q7: I think the presented information serves well as a useful online brochure.
Q5 Q6 Q7
Search 3.65 3.50 2.75
Percy 4.00 3.80 3.50
Post-system questionnaire – Understanding the intention of the user
• Q8: I would like to get more information on some specific projects presented.
• Q9: I would like to get more information because I have not got the needed information.
Q8 Q9
Search 3.95 3.15
Percy 3.90 2.90
Exit-questionnaire – Evaluating users’ preferences
Q1 Q2 Q3 Q4 Q5 Q6 Q7
Search 2.70 3.00 3.10 2.80 3.30 3.50 2.70
Percy 3.70 3.60 3.50 4.20 4.20 3.40 4.00
P < 0.01 0.03 0.04
focus o
n the s
earch
topic bett
er
make s
earch
task ea
sier
deliver
higher quali
ty info
attrac
t atte
ntion bett
er
provide bett
er ex
planati
on
provide more
compreh
ensiv
e info
prefere
nce
Summary of experimental results
• With Percy, subjects can better focus on their tasks.• With Percy, subjects understand better why a piece
of information is presented.• Subjects preferred Percy.
Discussion
• About tailored hypermedia(anything to say about the delivery method itself or limitations?)
– Delivery – who is in charge, user/author? or it is situation dependent? – How to apply AH to a dynamic resource (or even a different domain)
with minimum effort?– Is user’s expectation of AH system different from IR?
• About the evaluation– What to compare – Is it fair to compare the adaptive system with the
non-adaptive system? Or should we compare two adaptive systems? – Will the preference brings the enhancement of user performance? – (How to measure long term benefit?)