1
Topic 6:Usability Evaluation of IA
Applications and Mechanisms
Azene Zenebe, Ph.D.
Bin Mai, Ph.D.
Presentation Outline Introduction Usability of IA applications and mechanisms -
Reviewed Usability Evaluation: What, When and Why Usability Specification for Evaluation Usability Evaluation Methods
ˉ Analytical methods
ˉ Empirical methods Case Study Summary
2
3
Learning Objectives and Outcomes After completing this module, you should
be able to:
ˉ Describe the factors that affect usability of security systems
ˉ Describe the importance of evaluation of usability security systems
ˉ Prepare usability specification for evaluation
Learning Objectives and Outcomes (Continued)
ˉ perform usability evaluation or testing of a security system using an analytical method such as expert inspection
ˉ Perform usability evaluation or testing of a security system using an empirical method such as a field study or lab testing
ˉ Report results of usability evaluation as well as describe how the results can be used to make improvement
4
5
Introduction
Usability of IA application and mechanism - Reviewed ˉ Usability refers to the extent to which a product
can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of user - ISO 9241-11
6
Multi-dimensionality of Usability
Ease of learning Efficiency of use Memorability Effectiveness Error frequency and severity Subjective satisfaction
7
“Usable” Security Systemscan easily and quickly learn a security system that they have never seen to accomplish basic tasks can remember enough to use them later without major cost are able to effectively perform and successfully complete security tasks supported by them cannot make sever and frequent errors are satisfied with the interface and functions of the systems
8
Framework for studying usability of security systems
four principal components in a human-machine system ˉ TOOL ˉ USER ˉ TASK ˉ ENVIRONMENT
9
Definers provide the policies, guidelines, and standards
Builders are the real techies, who create and install security solutions
Administrators operate and administer the security tools
End-users include home users and employees who are novice to CISS
Four groups of people involved in Security systems
Usability Evaluation: What, When and Why
Usability evaluation: whether a security system is usable for the users
Goal of usability evaluation: identify and correct flaws associated with ease of use of a security system
Performed during design and testing (or post-implementation) phases
Evaluation is iterative – an ongoing process
10
11
Usability Specification for Evaluation
Usability specifications are statements of required usability characteristics that are precise and testable
Task analysis provides a more precise specification of what users are expected to do in order to accomplish a task successfully
A sample usability specification - authenticity of a website
12
Subtasks Usability Outcomes Expected
Displaying the digital certificate of the website
A user with at least …previous usage experience should be able to display the certificate in a 40 seconds or less, with no errors, and should rate ‘easy of finding the menu item/icon for displaying the certificate’ no less than 6 on a 7-point rating scale.
Determining if the website is authenticate or not
A user with at least …previous usage experience should be able to read and comprehend the certificate information in a 80 seconds or less, with no errors, and should rate ‘easy of use the menu item/icon for navigating the certificate’ no less than 6 on a 7-point rating scale.
13
Usability Evaluation Methods
Analytical Methods - conduct analysis of a system’s features with the respect to their impacts for use
Empirical Methods – collect and use data from a system’s users. It is also referred as user-based testing
14
Analytical Methods Expert’s knowledge stated as a
heuristic rulesˉ Ten Usability Heuristics by Jakob
Nielsenˉ Shneiderman’s 8 Golden Rules of
Interface Design
15
Empirical Methods
What usability evaluators want to know is what happens when users use the systemDifferent techniques are ˉ Field studies ˉ Usability Testing in a laboratory ˉ Controlled Experiments
16
Quick Quiz What are the main advantages and
disadvantages for analytical methods and empirical methods?
Come up with two sample scenarios in IA field where you think analytical methods should be preferred, and two other scenarios where you think empirical methods should be preferred
17
Steps for usability testing Identify and profile the representative users Select the setting Decide what tasks users should perform Decide how and what types of data to collect Perform necessary activities before test session Perform necessary activities during test session Perform necessary activities after test session
18
Usability Testing in a Laboratory
Validity concerns are associated with the following questions for lab based testingˉ Is the prototype system used in the testing
missing any important features ˉ Are test participants really the kind of users
who will use the system ˉ Will actual users do tasks like these
participants ˉ Will actual users be more distracted in their
offices
19
Using the Results of Usability Testing
Results need to be looked at and actionable information regarding usability problems and issues should be made for design teams
Provide recommendations to address the identified problems
20
Automated Usability Testing Tools
A List of 24 Web Site Usability Testing Toolsˉ http://www.usefulusability.com/24-usabi
lity-testing-tools/
UMD list of usability testing tools ˉ http://otal.umd.edu/guse/testing.html#se
ct3a
Jay Forbes’ presentation about usability testing tools ˉ http://www.gslis.utexas.edu/~l385t6rb/a
uto_tools.pdf
Quick Quiz
Suppose you are testing the usability of an IDS your company decided to implement. ˉ What will be the setting of the testing? ˉ Who will be the representative users?ˉ What type of data should you collect?
Justify your answers.
21
Quick Quiz
Among IT managers, business managers, usability specialists, or general public, who do you think are the main users for automated usability testing tools? Why?
What aspects of a usability study do you believe can never be automated? Why?
22
23
Case Perspectives: Usability Evaluation
Perspectives is a new approach to help clients securely identify Internet servers in order to avoid "man-in-the-middle" attacks
works with Firefox 3 extension Demo
Mission of the Perspectives detect whether a self-signed certificate is valid detect the fake security certificate attack and will
warn you
24
Usability Evaluation Design User Population
Potential Users: Novice, Intermediate and Expert in Security and IT
Targeted Users: Subset of the Potential Users Context of Uses
Using the Internet Home, free WiFi sites, and/or work Quite or Not Quite environment
Tasks: Banking, Shopping, etc.
25
Usability Evaluation DesignPerspectives: evaluating the authenticity of a public key based on accompanying signatures and making use of a Browser’s built-in mechanisms for such evaluation Requirements gathering
ˉDevelop usability specificationUsability Evaluation
ˉUsing InspectionˉUsing Empirical
26
Summary From this module, reader should take
away the following:ˉ Usability is a combination of factorsˉ Usability requires that users understands
the organization policy and rulesˉ There exist frameworks that guide the
usability evaluationˉ For different stakeholders, the goals of
usability differ
27
Summary (continued)From this module, reader should take away the following:
ˉUsability specification is required for usability evaluationˉThere are two categories of usability evaluation methodsˉThere existing some tools that automate usability testing
28
Discussion Topics
What are the advantages and disadvantages of Inspection method?
What are the advantages and disadvantages of Empirical method?
Compare and contrast the different methods of data collection. Describe the advantages and disadvantages of these methods.
29
Discussion Topics
How useful are these Heuristics for security systems? Which of the two is more relevant to security systems? Are these methods security systems dependent?
Is there a heuristics for security system interface design? Is there a methodology?
30
Discussion Topics
Describe and discuss scenarios where a system’s usability is important to one type of users, while not so important to another type
What are your opinions regarding the ideas that, as described by Jay Forbe, “automated usability testing is too good to be true”?
31
Project Ideas
Suppose your friend Joe opened an E-bay store online to sell his comic book collections, what data do you collect to evaluate his website’s usability?
Suppose a university Registrar Office hires you to evaluate the usability of its online registration system. What data would you collect?
32
Project Ideas
Prepare a sample usability specification built to track usability of a scenario for setting a firewall in Windows XP.
Develop a usability evaluation desing to track usability of an IDS (Intrusion Detection System)
Design a usability evaluation study for the latest release of PGP.
33
References 1. Braz, C. and Robert, J.-M. Security and usability: the case of the user authentication
methods. In Proceedings of the 18th International Conferenceof the Association Francophone d'Interaction Homme-Machine ACM, Montreal, Canada 2006 199-203
2. Garfinkel, S.L. Design Principles and Patterns for Computer Systems That Are Simultaneously Secure and Usable Department of Electrical Engineering and Computer Science, MASSACHUSETTS INSTITUTE OF TECHNOLOGY, Boston, 2005, 470.
3. Hoonakker, P., Bornoe, N. and Carayon, P., Password Authentication from a Human Factors Perspective: Results of a Survey among End-Users. In 3rd Annual Meeting of the Human Factors and Ergonomics Society, (San Antonio, TX, 2009).
4. Josang, A., Alfayyadh, B., Grandison, T., Alzomai, M. and Mcnamara, J., Security usability principles for vulnerability analysis and risk assessment. in Twenty-Third Annual In Computer Security Applications Conference, (Miami Beach, Florida, 2007), 269-278.
5. Lazar, J. Web Usability: A User-Centered Design Approach. Pearson, Addison Wesley, Boston, 2006.
6. Nielsen, J. Usability Engineering. Morgan Kaufmann, San Francisco, 1994.
34
References
7. Rosson, M.B. and Carroll, J.M. Usability Engineering: Scenario-based development of human-computer interaction. Morgan Kaufmann, San Francisco, 2002.
8. Shackel, B. Usability - Context, Framework, Definition, Design and Evaluation. in Richardson, S. ed. Human Factors for Informatics Usability, Cambridge University Press, Cambridge, 1991.
9. Shneiderman, B. and Plaisant, C. Designing the User Interface. Addison-Wesley, Boston, 2005.
10.Weir, C.S., Douglasa, G., Carruthers, M. and Jacka, M. User perceptions of security, convenience and usability for ebanking authentication tokens. Computer & Security, 28 (1-2). 47-62.
11.Whitman, M.E. and Mattord, H.J. Management of Information Security. Course Technology, Thomson Learning, Inc., Canada, 2004.
12.Whitten, A. and Tygar, D., Why Johnny can't encrypt? In USENIX, (1999).