UX from 30,000ft (COMP33512 - Lecture 15, 16 & 17 - 2012/2013)

Preamble Science and Generalisation Break! Evaluation Design and Analysis Practical Ethical Procedures Wrapping Up

The User Experience

from 30,000ft#comp33512

Week 08 – Lectures 15/16

(Thoughtworks)

Week 09 – Lecture 17

Simon Harper

University of Manchester

Semester 2 – 2012/13

last update: April 24, 2013

The User Experience from 30,000ft 1 / 22


UX Pop Quiz

1. Why is it difficult to know if the affective principles have

been captured in software correctly?

2. Why is affective computing different to affective experiences?

3. How do Aesthetics and Visual Complexity relate to each

other?

4. How does narrative art relate to the principle of Flow?

5. Why is Emotion difficult to quantify? What is one possible

solution?

The User Experience from 30,000ft Preamble 2 / 22

...expanded on pg. 186.


Designing Your Evaluations

1. Badly Designed = Incorrect Analysis;

2. Incorrect Analysis = Incorrect Conclusions; which means

3. Success of your Interventions in Doubt.

This Means

If evaluations are not designed correctly the previous ≈207 pages

of the course notes have been, to a large extent, pointless.


...expanded in ‘Designing Your Evaluations’ (pg. 207)






This Means










This Means










This Means






Science and Generalisation

Inductive reasoning

Evaluates and then applies to the general ‘population’

abstractions of observations of individual instances of members of

the same population

Deductive reasoning

The User Experience from 30,000ft Science and Generalisation 4 / 22




Inductive reasoning



the same population

Deductive reasoning

Evaluates a set of premises which then necessitate a conclusion –

for example: {(1) Herbivores only eat plant matter; (2) All

vegetables contain only plant matter; (3) All cows are herbivores}→ Therefore, vegetables are a suitable food source for Cows.





Inductive reasoning



the same population

Deductive reasoning

1. Therefore, the conclusion must be true provided that the

premises are true;

2. Note that you could not say ‘Therefore, all cows eat

vegetables’ because fruit also contains only plant matter; as

do grass and trees.




Scientific Bedrock

To Be scientific,

A method of inquiry must be based on the gathering of

observable, empirical and measurable evidence, and be subject to

specific principles of reasoning.


...expanded in ‘Scientific Bedrock’ (pg. 208)


Scientific Bedrock

Figure 77. ‘The Scientific Method’; pg. 208




Scientific Bedrock

An Inductive Example

1. Firstly, we create an hypothesis which, in the best case, cannot

be otherwise interpreted and is ‘refutable’; for example we might

make the statement ‘all swans are white’. In this case we may

have travelled widely and tried to observe swans in every country

and continent in an attempt to support our hypothesis.




Scientific Bedrock

An Inductive Example (pg. 208)

2. While, we may be able to amass many observations of white

swans we must also realise that a statement must be refutable. If

the hypothesis remains intact it must be correct; in our example

we may try to observe every swan that exists in, say, the UK, or

Europe, or the Americas, which is not white.




Scientific Bedrock

An Inductive Example (pg. 208)

3. However, one instance of an observation of a non-white swan

will disapprove our hypothesis; in this case when we arrive in

Australia we discover a black swan, in this case we can see all

swans are not white and our hypothesis is found to be incorrect.




Scientific Bedrock

I Many debates regarding the question of whether inductive

reasoning leads to truth;

I We can make some inductive leaps if they are based on good

science;

I These leaps may not be absolutely accurate; but

I May well assist us in our understanding; in the

I UX domain we use mathematical (statistical) methods to

help us understand these points.




Mathematical (Statistical) Methods

I Generalise results to enable us to say something about the

wider population; so

I We use well formed and tested statistical tests;

I Which enables use to mathematically generalise to a

population; this is called,

I External Validity.

No 100% Certainty

All we have is a level of confidence in how a particular test relates

to the population, and therefore how useful the knowledge

generated from it really is.




Mathematical (Statistical) Methods

I Generalise results to enable us to say something about the

wider population; so

I We use well formed and tested statistical tests;

I Which enables use to mathematically generalise to a

population; this is called,

I External Validity.

No 100% Certainty

All we have is a level of confidence in how a particular test relates

to the population, and therefore how useful the knowledge

generated from it really is.




Variables / UX

Behavioural: Equated to the user;

Stimulus: Equated to the interface or the computer

system;

Observable Response: the thing we measure to understand if

there is a benefit after we have

manipulated the stimulus; and

Subject: Factors such as age, weight, gender.

Independent Variable: The thing that we manipulate – the lower

the number of independent variables, the more

confident we can be about the data collected and

the results of the analysis; and

Dependent Variable: The thing that we measure.


...expanded in ‘Variables’ (pg. 209)


Measuring Dependent Variables

Nominal Scale: Which denotes identity;

Ordinal Scale: Which denotes identity and magnitude;

Interval Scale: Denotes identity, magnitude and has the benefit of

equal intervals; and

Ratio Scale: Which has the positive properties of the three we

have already seen as well as a true zero point.

Variables, and their measurement, are important.

They inform the experimental design process and the kind of

analysis that will be possible once the data has been collected.


...expanded in ‘Measuring Variables’ (pg. 210)


Measuring Dependent Variables

Nominal Scale: Which denotes identity;

Ordinal Scale: Which denotes identity and magnitude;

Interval Scale: Denotes identity, magnitude and has the benefit of

equal intervals; and

Ratio Scale: Which has the positive properties of the three we

have already seen as well as a true zero point.

Variables, and their measurement, are important.

They inform the experimental design process and the kind of

analysis that will be possible once the data has been collected.


...expanded in ‘Measuring Variables’ (pg. 210)


Hypothesis Testing

Null Hypothesis: Which dictates that there is no difference

between two conditions beyond chance

differences; or

Hypothesis: Which dictates there is a difference and

supports the hypothesis proposed.

Strong and Weak

A hypothesis must be ‘strong’ to be testable.

Nothing is Ever Proved

Hypotheses are supported or disproved - NOT ever proved (in

empirical work).. Why?


...expanded in ‘Hypothesis Testing’ (pg. 211)


Hypothesis Testing



differences; or



Strong and Weak








Hypothesis Testing



differences; or



Strong and Weak








Let’s Have a Break!

Back in 10 Minutes!

Come see me now if you have

Questions Regarding this Lecture!

The User Experience from 30,000ft Break! 11 / 22


Evaluation Design and Analysis

I Experimental Design;

I Data Collection and Tools;

I Data Analysis; mostly

I Statistical Analysis.

The User Experience from 30,000ft Evaluation Design and Analysis 12 / 22

...expanded in ‘Evaluation Design and Analysis’ (pg. 212)


Evaluation Design and Analysis

I Descriptive Statistics;

I Inferential Statistics.

I Internal Validity;

I External Validity; and

I Confounding Variables.


...expanded in ‘Evaluation Design and Analysis’ (pg. 212)


Participants

Simple Random Sampling Probabilistic — Simple random

sampling equates to drawing balls at a tom-bola. The selection of

the first has no bearing, and is fully independent of, the second or

the third, and so forth. This is often accomplished in the real

world by the use of random number tables or, with the advent of

computer technology, by random number generators;

Systematic Sampling Probabilistic — Systematic samples are a

variation of random sampling whereby each possible participant is

allocated a number, with participants being selected based on

some systematic algorithm. In the real world we may list

participants numbering them from, say, one to three hundred and

picking every seventh participant, for instance;


...expanded in ‘Participants’ (pg. 216)


Participants

Stratified Sampling Probabilistic — Stratified samples are used

to reduce the normal sampling variation that is often introduced

in random sampling methods. This means that certain aspects of

the sample may become apparent as that sample is selected. In

this case, subsequent samples are selected based on these

characteristics, this means that a sample can be produced that is

more likely to look like the total population than a random sample;

Multistage Sampling Probabilistic — Multistage sampling is a

strategy for linking populations to some kind of grouping. If a

sample was drawn from, say, the U. of Manchester then this may

not be representative of all universities. In this case, multistage

sampling could be used whereby a random sample is drawn from

multiple different universities independently and then integrated.

In this way we can ensure the generalisability of the findings; and




Participants

Quota Sampling Non-Probabilistic — Almost all

non-governmental polling groups or market research companies

rely heavily on non-probability sampling methods; the most

accurate of these is seen to be quota based sampling. Here, a

certain demographic profile is used to drive the selection process,

with participants often approached on the street. In this case, a

certain number of participants are selected, based on each point

in the demographic profile, to ensure that an accurate

cross-section of the population are selected;

Snowball Sampling Non-Probabilistic — The process of snowball

sampling is much like asking your participants to nominate

another person with the same trait as them.




Participants

Convenience Sampling Non-Probabilistic — The participants

are selected just because they are easiest to recruit for the study

and the UX’er did not consider selecting participants that are

representative of the entire population.

Judgmental Sampling Non-Probabilistic — This type of

sampling technique is also known as purposive sampling and

authoritative sampling. Purposive sampling is used in cases where

the specialty of an authority can select a more representative

sample that can bring more accurate results than by using other

probability sampling techniques.




Evaluation‘++’

I Single Group, Post Test;

I Single Group, Pre-Test and Post-Test;

I Natural Control Group, Pre-Test and Post-Test;

I Randomised Control Group, Pre-Test and Post-Test;

I Within Subjects; but there are

I Others.


...expanded in ‘Evaluation‘++’’ (pg. 219)


Practical Ethical Procedures

The Ethical Process

A critical component of good evaluation design because it

encourages the UX specialist to focus on the methodology and

the analysis techniques to be used within that methodology.

The User Experience from 30,000ft Practical Ethical Procedures 16 / 22

...expanded in ‘Practical Ethical Procedures’ (pg. 220)


Organisations

I The American Psychological Association’s (APA), ‘Ethical

Principles of Psychologists and Code of Conduct’;

I The United States Public Health Service Act (Title 45, Part

46, Appendix B), ‘Protection of Human Subjects’;

I The Belmont Report, ‘Ethical Principles and Guidelines for

the Protection of Human Subjects of Research’;

I The Council of International Organisations of Medical

Sciences, ‘International Ethical Guidelines for Epidemiological

Studies’; and finally

I The World Medical Association’s, ‘Declaration of Helsinki –

Ethical Principles for Medical Research Involving Human

Subjects’.




in Brief...

About You...

Competence: Keep up to date, know your limitations, ask for

advice;

Integrity: Have no axe to grind, or desired outcome; and

Science: Follow the Scientific Method.




in Brief...About Them...

Respect: Assess you participants autonomy and capability

of self-determination, treat participants as equals,

ensure their welfare;

Benefits: Maximising benefits and minimising possible harms

according to your best judgement, seek advice

from your organisations ethics committee;

Justice: Research should be undertaken with participants

who will benefit from the results of that research;

and

Trust: Maintain trust, anonymity, confidentiality and

privacy, ensure participants fully understand their

roles and responsibilities and those of the

experimenter.




in Brief...

About Us...

Responsibility: You have a duty of care, not only to your

participants, but also to the community from

which they are drawn, and your own community of

practice.




Discussion Topics Coursework # 3

‘Voice Loops as Cooperative Aids in Space Shuttle Mission

Control’ (10 Marks) – this paper shows just how far UX and the

techniques which it inherits from human computer interaction can

go. We are mainly concerned with systems and objects which are

purely commercial, however, in this case failures in the human

interface can have serious consequences for a real-time mission,

including the loss of the vehicle. Further, these kind of UX

techniques can also be found in other critical interface

components such as those controlling nuclear power stations or

fly-by-wire aircraft.Jennifer C. Watts, David D. Woods, James M. Corban, Emily S. Patterson, Ronald L. Kerr, and LaDessa C.Hicks., Voice loops as cooperative aids in space shuttle mission control., In Proceedings of the 1996 ACMconference on Computer supported cooperative work, CSCW ’96, pages 48–56, New York, NY, USA, 1996.ACM., ISBN 0-89791-765-0., http://doi.acm.org/10.1145/240080.240188., URLhttp://doi.acm.org/10.1145/240080.240188.

The User Experience from 30,000ft Wrapping Up 19 / 22

...expanded in ‘Discussion Topics’ (pg. 19)

http://doi.acm.org/10.1145/240080.240188

http://doi.acm.org/10.1145/240080.240188


Pop Quiz for next (logical) week...

1. What is the scientific method and why is it important?

2. What do we mean by internal and external validity?

3. What is the single most important reason for having a set of

ethical procedures?

4. What are the eight key ethical principles (give a brief

rationale for each)?

5. Why is conforming to scientific principles key to good ethical

designs?


...expanded on pg. 231.


To Do for next week...

1. Pop Quiz (pg. 231) Discuss Next Week; and

2. Read your notes up to the end of ‘Self Assessment

Questions’ (pg. 231)



Any Questions?

Simon Harper 2.44 Kilburn Building

0161 275 0599 (OR x50599)

[email protected]

Office Hours: Friday 14:00–18:00

Figure 93. ‘Simon Harper –Your Mild-Mannered CourseTutor’; pg. 326


...expanded in ‘Contact’ (pg. 326)

[email protected]

UX from 30,000ft (COMP33512 - Lecture 15, 16 & 17 - 2012/2013)

Education

preamble science

ft science

ft preamble

generalisation break

evaluation design

populationthe user experience

meansthe user experience

incorrect conclusions