SSIi2016 keynote Martijn Willemsen

Combining

process data

and subjective

data to better

understand

online behavior

Martijn Willemsen

Human-Technology

Interaction

• At the start of the century (2000ish…)

• From paper to computer-based experiments

• Running stuff online:

– Gathering additional measures

– Reading time per page

– Clicking patterns

– About 5-10% of data is invalid!

We should measure time in controlled Lab experiments too!

– 2003-2004

Process tracing

research

with Eric Johnson

3

MouselabWEBOnline process tracing tool to measure decision processes

4

1988

2004/2008

www.mouselabweb.org

• Tradeoff between Target and Competitor

– Price versus Quality

• Adding 3rd option: Decoy Da to TC set

• D is dominated by target T but not by

competitor C (and hardly ever chosen)

• P(T;DTC) > P(T;TC)

• Violation of independence of irrelevant

alternatives

Attraction Effect

TC DTC

T 46% 53%

C 54% 47%

6

Movie Attraction: subject 5384Direct impact of the decoy (DTC order, price first)

D T C

• Using Icon Graphs to plot the process data

• Dynamics:

– Scanning Phase (all acquisitions until all boxes have been

opened once)

– Choice phase (all remaining acquisitions)

– For Choice of target and not

8

Combining process with

subjective data

The case of recommender systems

9

• Iyengar and Lepper (2000): jam-study

• Apparently, satisfaction is not only a

function of attractiveness but also

of the choice difficulty

Choice overload

More attractive3% sales

Less attractive30% sales

Higher purchasesatisfaction

Using a movie recommender

Top5 1 2 3 4 5 - - - - - - - - - - - - - - -

Top20 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Lin20 1 2 3 4 5 99 199 299 399 499 599 699 799 899 999 1099 1199 1299 1399 1499

Results

perceived recommendation

varietyperceived recommendation

quality

Top-20vs Top-5 recommendations

movie

expertisechoice

satisfaction

choice

difficulty

+

+

+

+

-+

.401 (.189)p < .05

.170 (.069)p < .05

.449 (.072)p < .001

.346 (.125)p < .01

.445 (.102)p < .001

-.217 (.070)p < .005

Objective System Aspects (OSA)

Subjective System Aspects (SSA)

Experience (EXP)

Personal Characteristics (PC)

Interaction (INT)

Lin-20vs Top-5 recommendations

+

+ - +

.172 (.068)p < .05

.938 (.249)p < .001

-.540 (.196)p < .01

-.633 (.177)p < .001

.496 (.152)p < .005

-0.1

0

0.1

0.2

0.3

0.4

0.5

Top-5 Top-20 Lin-20

Choice satisfaction

•Median Choice rank

•Top 20: 8.5

•Lin 20: 3.0

•Looking time per item:

•Top 20: 2.8 sec

•Lin 20: 1.4 sec

•Acq. Freq per item:

•Top 20: .64

•Lin 20: .44

Frequency

Time

Behavioral data

14

Psychologists and HCI people are mostly interested in experience…

User-Centric Evaluation Framework

15

Computers Scientists (and marketing researchers) would study

behavior…. (they hate asking the user or just cannot (AB tests))


16

Though it helps to triangulate experience and behavior…


17

Our framework adds the intermediate construct of perception that explains

why behavior and experiences changes due to our manipulations


18

And adds personal

and situational

characteristics

Relations modeled

using factor analysis

and SEM

Knijnenburg, B.P., Willemsen, M.C., Gantner, Z., Soncu, H., Newell, C. (2012). Explaining

the User Experience of Recommender Systems. User Modeling and User-Adapted

Interaction (UMUAI), vol 22, p. 441-504

http://bit.ly/umuai


• Two cases that clearly shows the importance of the triangulation of

Behavioral data & Subjective data!

• Video recommender service: satisfaction versus clicks and

viewing times

• Diversification: continuing the choice overload work

– Can Diversification reduce choice overload?

– Choice difficulty: effort versus cognitive difficulty

19

20

Video Recommender system: EMIC Pre-trial in UMUAI paper

Knijnenburg, B.P., Willemsen, M.C. & Hirtbach, S. (2010). Receiving recommendations and providing feedback :the user-experience of a recommender system. E-Commerce and Web Technologies (11th InternationalConference, EC-Web 2010, Lecture Notes in Business Information Processing, Vol. 61, pp. 207-216)

21

• Diversification and list length as two experimental factors

– list sizes: 5 and 20

– Diversification: none (top 5/20), medium, high

• Dependent measure: choice satisfaction

– Choice difficulty versus attractiveness

– Subjective choice difficulty (scale) and objective choice difficulty (effort: hovers)

• 159 Participants from an online database

– Rating task to train the system (15 ratings)

– Choose one item from a list of recommendations

– Answer user experience questionnaire

Diversification & Choice Satisfaction

• Perceived recommendation diversity

– 5 items, e.g. “The list of movies was varied”

• Perceived recommendation attractiveness

– 5 items, e.g. “The list of recommendations was attractive”

• Choice satisfaction

– 6 items, e.g. “I think I would enjoy watching the chosen movie”

• Choice difficulty

– 5 items, e.g.: “It was easy to select a movie”

Questionnaire-items

Structural Equation Model

• Perceived Diversity increases with Diversification

– Similarly for 5 and 20 items

– Perc. Diversity increases attractiveness

• Perceived difficulty goes down with diversification

• Effort (behavioral difficulty) goes up with list length

• Perceived attractiveness goes up with diversification

• Diverse 5 item set excels…

– Just as satisfying as 20 items

– Less difficult to choose from

– Less cognitive load…!

-0.5

0

0.5

1

1.5

none med highstan

dar

diz

ed s

core

diversification

Perc. Diversity

5 items

20 items

-0.2

0

0.2

0.4

0.6

0.8

1

none med highst

and

ard

ized

sco

re

diversification

Choice Satisfaction

5 items20 items

• Behavioral and subjective data are two parts of the same story: you often need both to really get it!

• Try to capture as much of the process as you can, using smart interface designs, event tracking (hovers, clicks) or even cooler stuff such as modern cheap eye trackers (Tobii EyeX, EyeTribe)

• User-centric framework allows us to understand WHY particular approaches work or not– Concept of mediation: user perception

helps understanding..

What you should take away…

Contact:

Martijn Willemsen

@MCWillemsen

[email protected]

www.martijnwillemsen.nl

Thanks to my co-authors:

Mark Graus

Bart Knijnenburg

Dirk Bollen

Eric Johnson

Questions?

mailto:[email protected]

http://www.martijnwillemsen.nl/

SSIi2016 keynote Martijn Willemsen

Presentations & Public Speaking