User Engagement: from Sites to a Network of Sites or The Network Effect Matters!

- 1 -

User Engagement: from Sites to a Network of Sites

or The Network Effect Matters!

Ricardo Baeza-Yates Mounia Lalmas

Yahoo! Labs Barcelona

Joint work with Janette Lehmann and Elad Yom-Tov and many others at Yahoo! Labs

- 2 -

Outline

o  Motivation, definition and scope

o  Models of user engagement

o  Networked user engagement

- 3 -

Motivation, Definition and Scope

o  Definition and scope

o  Characteristics of user engagement

o  Measures of user engagement

o  Our research agenda

- 4 -

User Engagement – connecting three sides

Quality of the user experience that emphasizes the positive aspects of the interaction, and in particular the phenomena associated with users wanting to use a web application longer and frequently.

Successful technologies are not just used, they are engaged with

user feelings: happy, sad, excited, bored, …

The emotional, cognitive and/or behavioural connection that exists, at any point in time and over time, between a user and a technological resource

user interactions: click, read comment, recommend, buy, …

user mental states: concentrated, challenged, lost, interested …

S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper), WSDM Workshop on User Modelling for Web Applications, 2011.

- 5 -

Would a user engage with this web site?

http://www.nhm.ac.uk/

- 6 -

Would a user engage with this web site?

http://www.amazingthings.org/ (art event calendar)

- 7 -

Would a user engage with this web site?

http://www.lowpriceskates.com/ (e-commerce – skating)

- 8 -

Would a user engage with this web site?

http://chiptune.com/ (music repository)

- 9 -

Would a user engage with this web site?

http://www.theosbrinkagency.com/ (photographer)

- 10 -

Characteristics of user engagement (I)

• Users must be focused to be engaged • Distortions in the subjective perception of time used to

measure it Focused attention

• Emotions experienced by user are intrinsically motivating •  Initial affective hook can induce a desire for exploration, active

discovery or participation Positive Affect

• Sensory, visual appeal of interface stimulates, promote focused attention

• Linked to design principles (e.g. symmetry, balance, saliency) Aesthetics

• People remember enjoyable, useful, engaging experiences and want to repeat them

• Reflected in e.g. the propensity of users to recommend an experience/a site/a product

Endurability

- 11 -

Characteristics of user engagement (II)

•  Novelty, surprise, unfamiliarity and unexpected •  Appeal to user curiosity, encourages inquisitive

behavior and promotes repeated engagement Novelty

•  Richness captures the growth potential of an activity •  Control captures the extent to which a person is able

to achieve this growth potential Richness and control

•  Trust is a necessary condition for user engagement •  Implicit contract among people and entities which is

more than technological

Reputation, trust and expectation

•  Difficulties in setting up “laboratory” style experiments Motivation, interests,

incentives, and benefits

- 12 -

Forrester Research – The four I’s

•  Presence of a user •  Measured by e.g. number of visitors, time spent Involvement

•  Action of a user •  Measured by e.g. CTR, online transaction, uploaded

photos or videos Interaction

•  Affection or aversion of a user •  Measured by e.g. satisfaction rating, sentiment

analysis in blogs, comments, surveys, questionnaires Intimacy

•  Likelihood a user advocates •  Measured by e.g. forwarded content, invitation to join Influence

Measuring Engagement, Forrester Research, June 2008

- 13 -

Peterson et al Engagement measure - 8 indices Click Depth Index: page views Duration Index: time spent Recency Index: rate at which users return over time Loyalty Index: level of long-term interaction the user has with the site or product (frequency) Brand Index: apparent awareness of the user of the brand, site, or product (search terms) Feedback Index: qualitative information including propensity to solicit additional information or supply direct feedback Interaction Index: user interaction with site or product (click, upload, transaction)

Peterson etal. Measuring the immeasurable: visitor engagement, WebAnalyticsDemystified, September 2008

- 14 -

Measuring user engagement

Measures   Characteristics  

Self-reported engagement  

Questionnaire, interview, report, product reaction cards  

Subjective, user study (lab/online)

Mostly qualitative

Cognitive engagement  

Task-based methods (time spent, follow-on task)

Neurological measures (e.g. EEG)

Physiological measures (e.g. eye tracking, mouse-tracking)  

Objective, user study (lab/online)

Mostly quantitative

Scalability an issue?

Interaction engagement  

Web analytics + “data science” (CTR, bounce rate, dwell time, etc)

Metrics and user models  

Objective, data study

Quantitative Large scale

- 15 -

Interaction engagement – Online metrics Proxy of user engagement

- 16 -

Diagnostic and what we can do

Diagnostic: work exists, but fragmented. In particular: o What and how to measure depend on services and goals o Going beyond site engagement

What we have done: 1.  Models of user engagement 2.  Networked user engagement 3.  Complex networks analysis

Future: Economic model for networked UE

- 17 -

Models of User Engagement Online sites differ concerning their engagement!

Games Users spend much time per visit

Search Users come frequently and do not stay long

Social media Users come frequently and stay long

Special Users come on average once

News Users come periodically

Service Users visit site, when needed

is it possible to model these differences and compare different classes of sites?

- 18 -

Data and Metrics

Interaction data, 2M users, July 2011, 80 USA sites Popularity #Users Number of distinct users

#Visits Number of visits

#Clicks Number of clicks

Activity ClickDepth Average number of page views per visit.

DwellTimeA Average time per visit

Loyalty ActiveDays Number of days a user visited the site

ReturnRate Number of times a user visited the site

DwellTimeL Average time a user spend on the site.

- 19 -

Diversity in User Engagement

Users and Loyalty Sites have different user groups

Proportion of user groups is site-dependent

Time and Popularity Site engagement can be periodic or

contains peaks

Engagement of a site depends on users and time

mail, social media

shopping, entertainment

media (special events)

daily activity, navigation

media, entertainment

- 20 -

Methodology

General models User-based models Time-based models Dimensions

8 metrics 5 user groups 8 metrics per user group

weekdays, weekend 8 metrics per time span

#Dimensions 8 40 16

Kernel k-means with Kendall tau rank correlation kernel

Num. of clusters based on eigenvalue distribution of kernel matrix Significant metric values with Kruskal-Wallis/Bonferonni

#Clusters (Models) 6 7 5

Analyzing cluster centroids = models

- 21 -

Models of user engagement (I)

•  6 general models

•  Popularity, activity and loyalty are independent from each other

•  Popularity and loyalty are influenced by external and internal factors

  e.g. frequency of publishing new information, events, personal interests

•  Activity depends on the structure of the site

Models based on engagement metrics

interest-specific

e-commerce, configuration

periodic media

- 22 -

Models of user engagement (II)

User-based [7 models] Models based on engagement per user

group

Time-based [5 models] Models based on engagement over

weekdays and weekend

Models based on engagement metrics, user and time

navigation game, sport hobbies, interest-specific

daily news

  Sites of the same type (e.g. mainstream media) do not necessarily belong to the same model

  The groups of models describe different aspects of engagement, i.e. they are independent from each other

- 23 -

Relationships between models

General User Time General 0.00 3.50 4.23 User 3.50 0.00 4.25 Time 4.23 4.25 0.00

 Groups of models are independent from each other

 Example:  Model mu2

[high popularity and activity in all user groups, increasing loyalty]

 50% to model mt2 [high popularity on weekends and high loyalty on weekdays]

 50% to model mt3 [high activity and loyalty on weekends]

Variance of Information [0,5.61]

- 24 -

Recap & Next

User engagement is complex and standard metrics capture only a part of it

User engagement depends on users and time First step towards a taxonomy of models of user

engagement … and associated metrics

Next Interaction between models

User demographics, time of the day, geo-location, etc

J. Lehmann, M. Lalmas, E. Yom-Tov and G. Dupret. Models of User Engagement, UMAP 2012.

- 25 -

Understanding the problem: Users on Yahoo! network of sites

- 26 -

Networked user engagement: engagement across a network of sites

Large online providers (AOL, Google, Yahoo!, etc.) offer not one service (site), but a network of services (sites)

Each service is usually optimized individually, with some effort to direct users between them

 Success of a service depends on itself, but also on how it is reached from other services (user traffic)

Measuring user engagement across a network of sites should account for user traffic

between sites.

- 27 -

Online multi-tasking

users spend more and more of their online session multi-tasking, e.g. emailing, reading news, searching for information ONLINE MULTI-TASKING navigating between sites, using browser tabs, bookmarks, etc seamless integration of social networks platforms into many services

leaving a site is not a “bad thing!”

- 28 -

Online multi-tasking

Users switch between sites within an online session (several sites are visited and the same site is visited several times)

Navigation

Back button

Link Other1

1995 35.7% 45.7% 18.6%

1997 31.7% 43.4% 24.9%

2006 14.3% 43.5% 42.2% Oberdorf et al

1)  Usage of tabs, bookmarks, typing the URL directly,… 2)  http://uxmovement.com/navigation/why-external-links-should-open-in-new-tabs/

Browser usage changed Less usage of back button

- 29 -

Online multi-tasking

Users switch between sites within an online session (several sites are visited and the same site is visited several times)

Navigation

Back button

Link Other1

1995 35.7% 45.7% 18.6%

1997 31.7% 43.4% 24.9%

2006 14.3% 43.5% 42.2%

Oberdorf et al]

Dubroy et al

Browser usage changed More and more usage of tabs

1)  Usage of tabs, bookmarks, typing the URL directly,… 2)  http://uxmovement.com/navigation/why-external-links-should-open-in-new-tabs/

UX Movement2: External links affect your site and users Links that take users to different websites should open in new tabs.

- 30 -

Online multi-tasking

Users switch between sites within an online session (several sites are visited and the same site is visited several times)

 A short visit does not mean less engagement

Measuring user engagement across a network of sites should account for multi-tasking

Navigation

Back button

Link Other1

1995 35.7% 45.7% 18.6%

1997 31.7% 43.4% 24.9%

2006 14.3% 43.5% 42.2%

- 31 -

Networked user engagement - Two studies

o  Is there a network effect? o  study of 50 Yahoo! sites

o  downstream engagement as a measure of networked user engagement

o  effect of stylistics (layout and structure)

o  Can we quantify the network effect? o  study of 728 Yahoo! sites and traffic between them

o  use metrics from the complex network area together with engagement metrics to characterize networked user engagement

- 32 -

Is there a network effect?

o  Can we measure the network effect? o  Downstream engagement

o  Can we influence downstream engagement? o  Session types

o  Link types

The success of a web site largely depends on itself, but also on the network effect

This is particularly relevant for the Yahoo! network of properties

- 33 -

Measuring networked user engagement: Downstream engagement

User session

Yah

oo! s

ites

Downstream engagement for site A

(% remaining session time)

Site A

- 34 -

Downstream engagement

Varies significantly across sites

Exhibits different distributions according to site type

Is not highly correlated with other engagement measures such as dwell time

Optimizing downstream engagement will have little effect on user engagement within that site

0% 10% 20% 30% 40% 50% 60% 70%

0

0.02

0.04

0.06

0.08

0.1

0.12

1%

9%

17%

25

%

33%

41

%

49%

58

%

66%

74

%

82%

90

%

98%

0 20 40 60 80

100 120 140

0% 20% 40% 60% 80%

- 35 -

What causes engagement to change?

Web page style?

Web page content?

- 36 -

Methodology

Front pages of 50 popular Y! properties crawled every

hour

Sample user data from

toolbar data (19.4M

sessions, 265K users)

May 2011

Defining downstream engagement

measure

Measuring downstream engagement measure with

web page stylistics and

content

Influencing network

downstream engagement

with front page

stylistics

Studying the effect of links

on front pages on

downstream engagement

- 37 -

Data

Page attributes were defined for 50 popular (by page views) Yahoo! sites: Sampled the front page once every hour during the month of May 2011. Generate two types of attributes for each site at each time and date:

Stylistics (layout and structure) of a page General such as time of day, date, weekday or not

Downstream engagement values were measured using Yahoo! toolbar data: A total of 19.4M sessions recorded from approximately 265,000 users.

User and front page datasets joined by site, date and time, such that for each site and each date and time combination we have: average downstream engagement average dwell time vector of corresponding style attributes collected around the time that user

engagement was measured

- 38 -

Page stylistics provide good information to predict downstream engagement for many Yahoo! sites

Accuracy Precision Average DE site 1 site 2 site 3 site 4 site 5 site 6 site 7 site 8 site 9 site 10

0.80 0.76 0.72 0.71 0.65 0.63 0.63 0.63 0.61 0.60

0.54 0.52 0.43 0.40 0.42 0.34 0.38 0.44 0.34 0.31

0.07 0.11 0.21 0.18 0.20 0.19 0.26 0.14 0.18 0.13

The top-10 sites for which downstream engagement (DE) could be “accurately” predicted based on their stylistics

Downstream engagement of a number of sites of not particular types (models) could not be predicted from their stylistics.

- 39 -

Influential features

o Time of day

o Number of (non-image/non-video) links to Yahoo! sites in HTML body o Average rank of Yahoo! links on page o Number of (non-image/non-video) links to non-Yahoo! sites in HTML body

o Number of span tags (tags that allow adding style to content or manipulating content, e.g. JavaScript)

o  Link  placements  and  number  of  Yahoo!  links  can  influence  downstream  engagement  

o  Not  new,  but  here  shown  to  hold  also  across  sites    

o  Links  to  non-­‐Yahoo!  sites  have  a  posi>ve  effect  on  downstream  engagement  

o  Possibly  because  when  users  are  faced  with  abundance  of  outside  links  they  decide  to  focus  their  aBen>on  on  a  central  content  provider,  rather  than  visi>ng  mul>tude  of  external  sites    

- 40 -

Number of unique Yahoo! links (-) Number of (non-image/non-video) links

to Yahoo sites in the body of the HTML (+)

Number of (non-image/non-video) links to non-Yahoo! sites in the body of the HTML (+)

Number of video links within the page (+) Number of Java scripts on the page (-)

Three case studies

Here we look at three different Yahoo! sites, and the effect of their stylistics for downstream engagement

Time of day (+) Weekend (-) Number of unique Yahoo! links (-) Average rank of Yahoo! links on page (-) Number of paragraph tags (+)

Number of image links to non-Yahoo! sites in the body of the HTML (-)

Number of table elements (-) Average rank of Yahoo! links on page (-) Number of (non-image/non-video) links

to non-Yahoo! sites in the body of the HTML (+)

Time of day (+)

Sites Average downstream engagement

Accuracy Precision

e-commerce news women-interests

0.26 (+/- 0.31) 0.15 (+/- 0.02) 0.21 (+/- 0.06)

0.63 0.65 0.72

0.38 0.37 0.53

- 41 -

Influencing engagement through links

e-commerce news women- interests

Downstream engagement Same site Other Y! site Non Y! site

0.03 -0.09 -0.10

-0.31 0.20 0.04

-0.27 0.22 -0.25

Dwell time Same site Other Y! site Non Y! site

0.51 -0.61 -0.51

0.78 0.38 0.04

0.82 -0.68 0.80

The correlations between the number of various links and the values of downstream engagement and dwell time

• e-commerce: • links have little effect on downstream engagement, but have on dwell time

• news: • more news stories lead to more time spent on news • external links do not affect downstream engagement, but affect dwell time

• women-interests: • links to other Yahoo! sites can help increase engagement, but they may decrease dwell time

- 42 -

Users are more amenable to enhancing downstream engagement during certain sessions

Goal-specific sessions are those sessions where users have a specific goal in mind: do email, read news, check FB

Sessions when at least 50% of visited sites belonged to the five most common sites (for that user) were classified as goal-specific Goal-specific sessions accounted for 38% of sessions.

Approximately 92% of users had sessions of both kinds.

Average downstream engagement in goal-specific sessions was 0.16 vs. 0.2 for other sessions.

Accuracy of predicting downstream engagement was 0.76 for goal-specific sessions vs. 0.81 for other sessions.

When users do not have specific goals in mind, they may be more ready to accept suggestions (e.g. more links) for additional browsing

- 43 -

Further Work: Quantifying the Network Effect

Previously using one metric (downstream engagement), we showed that there is a network effect, and that the network effect can be influenced.

We go one step further and propose a methodology to account for the traffic between sites when measuring user engagement on a network of sites.

o  Engagement networks

o  Metrics from complex networks area (network-level and node-level)

o  Application on 728 Yahoo! sites

- 44 -

Thank you

Thanks to many people at Yahoo! Labs

Questions?