Evaluating usability through claims analysis Suzette Keith Ann Blandford, Bob Fields, Richard Butterworth, Yin Leng Theng
Dec 20, 2015
Evaluating usability through claims analysis
Suzette KeithAnn Blandford, Bob Fields,
Richard Butterworth, Yin Leng Theng
2
Aims
• Testing claims analysis as an analytical usability evaluation technique for digital libraries
• Validating claims with user trials• Testing usability of claims analysis for
designers• … towards developing a suite of usable
usability tools for digital libraries
Claims analysis
“claims analysis is an analytical evaluation method involving the identification of scenario features that have significant positive and negative usability consequences” (Rosson, Carroll 2002)
Case studies
1. Exploring claims analysis with the
developers
2. Applying claims analysis to an add-on
feature
3. User trial to investigate claims
Claims analysis: overview of method
1 Create scenarios - users, goals, story, action
2 Develop description of the artifact3 Generate claims4 Analyse claims5 Reason about design rationale
Data sources:• Interviews with users• Cognitive processing model• Research on information seeking• Research on digital library usage…Natasha is a recent graduate investigating
a new project. She decides to try the digital library and finds some results…
1 Create scenarios
2 Description of the artifact
• Design rationale• Usability specification• Functional specification• Mock-ups• Scripts• Partial implementation• Prototyping…she sees the search-box and types in a
query…
3 Generate claims
The Action cycle:How does the artifact:• evoke goals in the user?• support planning?• indicate action to take?• guide interpretation?• help evaluation and error recovery?
Goal claim
...Natasha is seeking information on a topic to support on-going project activity..
Claim about users goals:The digital library holds appropriate
information accessed by keyword searching and browsing
4 Analyse goal claim
• User can search or browse for information
• but • May not know appropriate terms• Content may not be appropriate
5 Reasoning about design rationale
• What factors might cause the user to fail• How can the effect of negative claims be
moderated• What trade-offs might effect other parts
of the system design
Case study 1:Exploring claims
Findings from developers:• Developers had designed the ‘best possible’
system • Informal use of novice scenarios• Focus on problems of awareness and
exploratory use• Difficult to identify negative consequences
of design decisions
Case study 2:Applying claims
Add-on feature in development:
• Hypothetical novice user scenarios
• Prototype interface
• Developers wanted support for design decisions
• Analysis strategy: plan, execute and evaluate
15
From ‘add a URL to this information space’
Plan? Action? Evaluation?
Claim for evaluation
….Natasha enters details into a form, the web-site or document url is notified to all profiled users..
Implemented feature: automatic completion sequence:
+ rapid execution minimises user effortbut- reduced opportunity to correct errors, maximum
opportunity for public embarrassment
Reasoning about evaluation claim
• Trade-off between speed and accuracy• Importance of feedback and consequence
of error• Exploration of alternative scenarios• Examination of technical issues• Proposed changes to interface
Case study 3:Novice user trials
The ‘ Google generation’• Positive and negative effects of prior
experience on goal, planning, action, evaluation
• Very specific information seeking task• Difficulty dealing with very large results sets• Progress measured by relevant full-text
documents found
The challenge
…what is exciting about digital libraries is the power they offer ordinary users…
The promise that information can be stored,accessed, manipulated and moved in ways that were hereto impossible…(Dillon)
20
Claims analysis: reprise
Can claims analysis contribute to the understanding of information seeking and the design of digital libraries ?
- task-artifact cycleCan claims analysis be specified to support the development of innovative and creative solutions to usability issues?
- concrete yet flexible