Seminar in E-Business & Recommender Systems recommendations and suggestions for future work regarding the impacts of Recommender Systems on Business and Customers will be proposed.

Seminar in E-Business & Recommender Systems

University of Fribourg, Department of Informatics

Research Paper

The impact of Recommender Systems on Business and Customers in Electronic Markets.

STUDENT NAMES: José A. Mancera, Philipp Bosshard STUDENT NUMBERS: 10-801-207, 06-200-844 COURSE NAME: Electronic Business and Recommender Systems

DEPARTMENT: Department of Informatics

SUPERVISOR: ASSISTANT: DATE OF SUBMISSION:

Prof. Dr. Andreas Meier Luis Terán 05-03-2015

II

Table of contents

List of Figures ............................................................................................................. V

1. Introduction ............................................................................................................. 1

1.1 Problem statement ............................................................................................ 1

1.2 Research Objectives and Methodology ............................................................. 2

1.2.1 Research Questions ................................................................................... 2

1.2.2 Objectives and Output of the thesis ............................................................ 2

1.2.3 Research Methodology ............................................................................... 2

1.3 Timetable .......................................................................................................... 3

1.4 Addressees ....................................................................................................... 3

2. Recommender Systems ......................................................................................... 3

2.1 Overview on Recommender Systems ............................................................... 3

2.2 Recommender System Categories ................................................................... 5

2.2.1 Collaborative Recommendation.................................................................. 5

2.2.2 Content-Based Recommendation ............................................................... 5

2.2.3 Knowledge-Based Recommendation ......................................................... 6

2.2.4 Hybrid Recommendation ............................................................................ 7

2.3 Most popular recommendation techniques per category ................................... 8

3. Recommender System Algorithms ....................................................................... 10

3.1 Similarity Measures ......................................................................................... 10

3.2 Collaborative Filtering Algorithms.................................................................... 11

4. Impacts of Recommender Systems on Business and Users ................................ 16

4.1 Offline and Online Evaluation .......................................................................... 16

4.1.1 Offline Evaluation: .................................................................................... 16

4.1.2 Online Evaluation: .................................................................................... 16

4.2 Accuracy ......................................................................................................... 17

4.3 Impacts of Recommender Systems on Business ............................................ 17

4.4 Impacts of Recommender Systems on Users ................................................. 18

III

5. Evaluation of Recommender Systems and their value for Business and Users ... 21

5.1 Recommendation System Properties .............................................................. 21

5.1.1 Prediction Accuracy .................................................................................. 22

5.1.2 Coverage .................................................................................................. 23

5.1.3 Confidence ............................................................................................... 23

5.1.4 Trust ......................................................................................................... 23

5.1.5 Novelty ..................................................................................................... 24

5.1.6 Serendipity ............................................................................................... 24

5.1.7 Diversity .................................................................................................... 24

5.1.8 Utility ......................................................................................................... 24

5.1.9 Risk .......................................................................................................... 25

5.1.10 Robustness............................................................................................. 25

5.1.11 Privacy .................................................................................................... 25

5.1.12 Adaptivity ................................................................................................ 26

5.1.13 Scalability ............................................................................................... 26

5.2 Summary of the Recommendation System Properties .................................... 27

5.3 Evaluation Metrics for Recommender Systems .............................................. 27

5.3.1 Accuracy Metrics ...................................................................................... 28

5.3.2 Privacy Metrics ......................................................................................... 28

5.3.3 Adaptivity Metrics ..................................................................................... 29

5.3.4 Trust Metrics ............................................................................................. 29

5.3.5 Confidence Metrics ................................................................................... 29

5.3.6 Novelty Metrics ......................................................................................... 29

6. Correlation between evaluation metrics and impacts ........................................... 30

6.1 Documented Scenarios ................................................................................... 31

6.1.1 Impact of Accuracy on Online Time .......................................................... 31

6.1.2 Impacts of Accuracy, Privacy and Adaptivity on User Preferences .......... 33

6.1.3 Impact of Accuracy on Product Views ...................................................... 35

IV

6.2 Non-documented Scenarios ............................................................................ 37

6.2.1 Impacts of Trust on Product Sales and Satisfaction ................................. 37

6.2.2 Impacts of Novelty/Confidence on Product Sales and User Preferences . 38

7. Analysis and Results ............................................................................................ 40

7.1 Documented Scenarios ................................................................................... 40

7.2 Non-Documented Scenarios ........................................................................... 40

7.3 The correlation results between RS Properties Results and Impacts ............. 41

8. Recommendations ................................................................................................ 45

8.1 Variety in Recommender System Properties .................................................. 45

8.1.1 Non-documented scenarios suggestions .................................................. 45

8.2 Perception of the User .................................................................................... 46

8.3 New Research Questions ............................................................................... 46

9. Conclusion ............................................................................................................ 48

10. Future Work ........................................................................................................ 49

11. References ......................................................................................................... 50

V

List of Figures

Figure 1: Recommender System Categories .............................................................. 4

Figure 2: Collaborative Recommendation Techniques ............................................... 8

Figure 3: Content-Based Recommendation Techniques ............................................ 9

Figure 4: Knowledge-Based Recommendation Techniques ....................................... 9

Figure 5: Hybrid Recommendation Techniques: ......................................................... 9

Figure 6: Impacts of Recommender Systems on Business ...................................... 17

Figure 7: Impacts of Recommender Systems on Users ........................................... 18

Figure 8: Attributes for Consumer Perceptions [12] .................................................. 20

Figure 9: Recommendation System Properties ........................................................ 21

Figure 10: Accuracy Predictions ............................................................................... 22

Figure 11: Recommendation System Properties ...................................................... 27

Figure 12: Scope of our impacts analysis ................................................................. 30

Figure 13: The impact of Accuracy on Online Time .................................................. 31

Figure 14: Summary of user's tolerable waiting time for computer response ........... 32

Figure 15: The impact of Accuracy, Privacy and Adaptivity on User Preferences .... 33

Figure 16: The impact of Accuracy on Product Views .............................................. 35

Figure 17: Average Items viewed for 1 Control Group and 3 Treatment Groups ...... 36

Figure 18: The impact of Trust on Product Sales and Satisfaction ........................... 37

Figure 19: The impact of Novelty on Product Sales and User Intention ................... 38

Figure 20: Research Paper Categories in Recommender Systems Field ................. 41

Figure 21: Direction of the Recommender System Design ....................................... 43

Figure 22: Relations between Metrics, RS properties and Impacts on Documented

Scenarios .................................................................................................................. 43

Figure 23: Relations between Metrics, RS properties and Impacts on non-

documented Scenarios ............................................................................................. 45

Figure 24: Relation between RS Properties and their impact on Users and Business

................................................................................................................................. 47

1

1. Introduction

1.1 Problem statement

Overview

In this age full of data and big amounts of information, people use a variety of

strategies to make choices about what to buy, how to spend their leisure time, and

even what to eat. Recommender systems automate some of these strategies with the

goal of providing to the user affordable, personal, and high-quality recommendations

in order to help the user in the decision-making process. After all, how can one be

sure that the user is not being manipulated and what is the recommenders influence

power on the users and on the business?

Firstly, the objective of this seminar thesis is to determine the impacts that

Recommender Systems can have on Business and Users from the side of consumer

research and marketing. Secondly, documentation and analysis of different research

papers was carried on in order to identify the most important evaluation metrics for

Recommender Systems. Thirdly, the document shows the results or our analysis

based on the evidence found concerning the correlation between evaluation metrics

and impacts of RS.

Finally, recommendations and suggestions for future work regarding the impacts of

Recommender Systems on Business and Customers will be proposed.

As an important remark, recommender Systems have multiple areas of application

consequently the focus of this paper only lies on e-business and e-commerce

segments (e.g. Amazon, eBay and other kinds of e-shops) and the position of

Recommender Systems for other types of segments are not part of the scope of this

document.

2

1.2 Research Objectives and Methodology

1.2.1 Research Questions

The next group of questions is the guideline of our study, each of them is answered

in sequence during the evolution of the document.

1. What are the impacts of Recommender System on users?

2. What are the impacts of Recommender System on business?

3. What are the most important evaluation metrics that can be used to analyze

influences on the Users and the Business?

4. What is the correlation between the evaluation metrics of and the influence of

recommender systems on business and users

5. Which aspects have been neglected in previous research papers?

1.2.2 Objectives and Output of the thesis

Based on the research questions, the main objectives of this seminar research paper

analysis are to find potential correlations between impacts and recommender system

(RS) properties. In addition recommendations and suggested paths of research are

also suggested further in the document, in order that other researchers can continue

our work and find more evidence about the metrics, which it would be essential to

design, improve or interpret the output of recommender systems. Understanding

these correlations requires a profound understanding about what recommender

systems are?, How do they work? and what are the influences that they have on

online customers and business?

1.2.3 Research Methodology

In order to answer the questions of this research seminar paper, our analysis would

be based not only on data of previous studies in the field of Recommender Systems,

consumer research and marketing but also on selected handbooks or textbooks

recommended by our supervisors. All these documentation would guide us to

discover or identify the variables that play a main role in the impact on users and

business in the segments of e-business and e-commerce.

3

1.3 Timetable

02-27-2015 Acceptance of working title

03-06-2015 Submission of the proposal

March 2015 Continue literature Research and reading

Writing Chapter 1, 2 and 3

April 2015 Writing Chapter 4

Draft of the paper

Finishing the report

Revision and Correction

April 2015 Midterm Appointment

05-03-2015 Submission of the thesis report

05-08-2015 Presentation of the thesis report

1.4 Addressees

The target audience of this paper is primarily students in the fields of computer

science, e-commerce, marketing and professionals who are involved in the field of

Recommender Systems. The results of this seminar document should provide the

parties mentioned above not only valuable knowledge in order to better understand,

analyze and improve the quality of Recommender Systems, but also a better

understanding of the consequences of the algorithms on users and business.

2. Recommender Systems

In order to get a better understanding and interpretation of the findings presented in

this seminar paper, it is important to review some core concepts in the field of

recommender systems before moving to the analysis and results. In the next two

chapters, there is a briefly overview of recommender systems, characteristics and

description of the algorithms considered in our analysis.

2.1 Overview on Recommender Systems

The majority of the Internet users have experienced in their visits to online stores or

web pages that offer their services, certain kind of recommendations from other users

and suggestions after reviewing a product as “Customers who bought this Item also

bought,” or “Customers who read this book also read these”. These

recommendations are becoming more often in the context of e-Commerce.[6]

4

Among all the different definitions, a Recommender System (RS) is software and

techniques which provide suggestions that determine whichever articles should be

shown to a particular visitor. [6]

Every RS requires that the system knows something about every user and it must

maintain a user model or user profile which contains for instance the user

preferences and remember the activity of the user, in order to be able to predict the

articles that might be interesting for the user. The way the RS collects this information

depends on the particular Recommendation technique, user preferences can, for

instance, be acquired implicitly by monitoring user behavior but recommender system

might also explicitly ask the visitor about his or her preferences. Moreover, it is

important to collect additional information, like opinions and tastes of a large

community and not only individual approaches.[6]

The variety in the information that can be collected is very wide but the most

important is to know which information the system could exploit when it generates a

list of personalized recommendations. Figure 1 gives an overview of the four main

RS categories:

Figure 1: Recommender System Categories

Recommender System

Collaborative

Content-Based

Knowledge-Based

Hybrid

5

2.2 Recommender System Categories

The next subsections give a short walkthrough around these four different RS

categories in order to understand the characteristics and parameters that are

involved in each of them

2.2.1 Collaborative Recommendation

The core idea of this type of recommender system is that if users shared the same

interests in the past and for instance the users that viewed or purchased the same

books will also have similar interests or preferences in the future.

As an example considers the case when user A and user B have a purchase history

that is strongly similar and user A has recently bought a book that B has not yet seen,

the basic rationale is to propose this book also to B. Because this selection of

hopefully interesting books involves filtering the most promising ones from a large set

and because the users implicitly collaborate with one another, this technique is also

called collaborative filtering (CF). [6]

Techniques under this category are wide used in the context of e-Commerce, the

advantage of these techniques is that the recommender system does not need to

know what the commodity or product is about, its genre, or who create it. Based on

these conditions to propose commodities that are actually similar to those the user

liked in the past might be more effective. [6]

2.2.2 Content-Based Recommendation

The techniques related to this category are based on the availability of item

descriptions and a profile that assigns importance to these characteristics. For

instance, thinking in an online store, the possible characteristics of the products might

include the genre, the specific topic, or the author. Similar to item descriptions, user

profiles may also be automatically derived and “learned” either by analyzing user

behavior and feedback or by asking explicitly about interests and preferences. [6]

In the context of content-based recommendation, the techniques are focused on how

the systems automatically acquire and continuously improve user profiles and how do

they determine which items match, similar items or common interest among users. [6]

6

Content-based recommendation has two advantages:

1. It does not require large user groups to achieve reasonable recommendation

accuracy.[6]

2. New items can be immediately recommended once item attributes are

available. [6]

2.2.3 Knowledge-Based Recommendation

There are market segments where the products are highly sophisticated, for instance

the consumer electronics, which involves not only a large number of one time buyers

but also the customer, does not have all the knowledge to understand about these

technologies. [6]

These simple facts bring new problems such that the recommender system cannot

rely on the existence of a purchase history, build user profiles or propose products

that other users bought. In addition more detailed and structured content must be

considered as technical and quality features. [6]

The solution that brings an answer in this context is a system that exploits additional

and means–end knowledge to generate recommendations. In such knowledge-based

approaches, the recommender system typically makes use of additional, often

manually provided, information about both the current user and the available items.

[6]

As a simple example of techniques in this category are Constraint based

recommenders which for instance in the case of the digital camera domain, a

constraint-based system could use detailed knowledge about the features of the

cameras, such as resolution, weight, or price. In addition, explicit constraints may be

used to describe the context in which certain features are relevant for the customer,

such as, for example, that a high resolution camera is advantageous if the customer

is interested in printing large pictures. Moreover the system could ask the user about

the relative importance of features, such as whether resolution is more important than

weight in order to provide recommendations. Although there are many different

techniques in this category, the recommender system should be able to answer

questions as: [6]

What are the mechanisms that can be used to select and rank the items based

on the user’s characteristics? [6]

7

How do we acquire the user profile in domains in which no purchase history is

available, and how can we take the customer’s explicit preferences into

account? [6]

Which interaction patterns can be used in interactive recommender systems?

[6]

Finally, in which dimensions can we personalize the dialog to maximize the

precision of the user preferences? [6]

2.2.4 Hybrid Recommendation

The previous categories have shown the main philosophy behind each of

recommender systems main categories, nevertheless combining recommendation

techniques can offer a better solution and more precise recommendations to specific

problems and the advantages and disadvantages depends on the problem setting.

[6]

For instance, community knowledge exists and detailed information about the

individual items is available, a recommender system could be enhanced by

hybridizing collaborative or social filtering with content-based techniques. In

particular, such a design could be used to overcome the described increase of

problems with pure collaborative approaches and rely on content analysis for new

items or new users. Combining different approaches, the recommender systems

should be able to answer: [6]

Which techniques can be combined, and what are the prerequisites for a given

combination? [6]

Should proposals be calculated for two or more systems sequentially, or do

other hybridization designs exist? [6]

How should the results of different techniques be weighted and can they be

determined dynamically? [6]

Last but not least, the classification and understanding of the philosophy behind each

category shows a general overview and it is important to remember that these

techniques vary constantly. The next section shows a more detailed classification of

some of these particular techniques per category.

8

2.3 Most popular recommendation techniques per category

Although there are many recommendation techniques used currently in the market

and mentioning all of them, would be impossible. This report shows some of them

and as general overview in order to give a better picture to the reader about the

outlook of the recommenders systems. The next figures show some of the

techniques by category.

Figure 2: Collaborative Recommendation Techniques

Collaborative

User-based nearest

neighbor

Item-based nearest

neighbor

User–item ratings matrix

SVD-based recommend

ation

Content-based

Content representatio

n and content

similarity

Similarity-based

retrieval

Text classification

methods

9

Figure 3: Content-Based Recommendation Techniques

Figure 4: Knowledge-Based Recommendation Techniques

Figure 5: Hybrid Recommendation Techniques:

Knowledge-based

recommendation

Constraint-based

recommenders

Case-based recommend

ers

Hybrid recommendation approaches

Monolithic hybridization

design

Parallelized hybridization

design

Pipelined hybridization

design

10

3. Recommender System Algorithms

It is essential to mention that the number of recommender systems that are available

for applications or in development are uncountable and it would be very extensive to

treat all of them in a document, in this section for the purposes of our research, we

have selected some of them which would be supportive later to make a deeper

analysis of some particular scenarios.

3.1 Similarity Measures

Similarity measures are normally applied as preprocessing data tools, which prepare

the data in order to apply a recommender algorithm. Similarity Measures are

considered as a complement of most of the algorithms. The next section shows the

most relevant ones that are used in most of the research papers that were

analyzed.[4]

3.1.1 Pearson Correlation

The similarity between items can also be given by their correlation which measures

the linear relationship between objects. Given the covariance of data points x and y

Σ, and their standard deviation σ, we compute the Pearson correlation using

[4]

3.1.2 Cosine

This measure indicates vector dot product and ||x|| ,which is the norm of vector x.

This similarity is known as the cosine similarity or the L2 Norm. Where σ is the

covariance matrix of the data. Another very common approach is to consider items as

document vectors of an n-dimensional space and compute their similarity as the

cosine (x) of the angle that they form: The cosine-based approach defines the

cosine-similarity between two users x and y as:[4]

11

3.1.3 Euclidean distance:

The simplest approach to measure similarity is the Euclidean distance, where d(x,y)

is the degree of the distance:

Where n is the number of dimensions (attributes) and 𝑥𝑘 and 𝑦𝑘 are the kth attributes

(components) of data objects x and y, respectively [6]

3.2 Collaborative Filtering Algorithms

During the seminar research, we found that most of the algorithms applied in different

research paper cases were mostly collaborative filtering algorithms. In this subsection

we mention the three main categories and the description of a particular algorithm

that was cited several times in different papers, k-Nearest Neighbors algorithm.

3.2.1 Item-based CF

Collaborative filtering (CF) is a technique used by some recommender systems.

Collaborative filtering has two senses, a narrow one and a more general one. In

general, collaborative filtering is the process of filtering for information or patterns

using techniques involving collaboration among multiple agents, viewpoints, data

sources, etc. Applications of collaborative filtering typically involve very large data

sets. Collaborative filtering methods have been applied to many different kinds of

data including: sensing and monitoring data, such as in mineral exploration,

environmental sensing over large areas or multiple sensors; financial data, such as

financial service institutions that integrate many financial sources; or in electronic

commerce and web applications where the focus is on user data, etc. The remainder

of this discussion focuses on collaborative filtering for user data, although some of

the methods and approaches may apply to the other major applications as well. [6]

In the newer, narrower sense, collaborative filtering is a method of making automatic

predictions (filtering) about the interests of a user by collecting preferences or taste

information from many users (collaborating). The underlying assumption of the

collaborative filtering approach is that if a person A has the same opinion as a person

B on an issue, A is more likely to have B's opinion on a different issue x than to have

12

the opinion on x of a person chosen randomly. For example, a collaborative filtering

recommendation system for television tastes could make predictions about which

television show a user should like given a partial list of that user's tastes (likes or

dislikes).Note that these predictions are specific to the user, but use information

gleaned from many users. This differs from the simpler approach of giving an

average (non-specific) score for each item of interest, for example based on its

number of votes. [4] [6]

3.2.2 Memory-based CF

This approach uses user rating data to compute the similarity between users or

items. This is used for making recommendations. This was an early approach used in

many commercial systems. It is effective and easy to implement. Typical examples of

this approach are neighborhood -based CF and item-based/user-based top-N

recommendations. For example, in user based approaches, the value of ratings user

'u' gives to item 'i' is calculated as an aggregation of some similar users' rating of the

item: [4] [6]

Where 'U' denotes the set of top 'N' users that are most similar to user 'u' who rated

item 'i'. Some examples of the aggregation function includes:

Where k is a normalizing factor defined as and is the average

rating of user u for all the items rated by u. [4] [6]

The neighborhood-based algorithm calculates the similarity between two users or

items, produces a prediction for the user by taking the weighted average of all the

ratings. Similarity computation between items or users is an important part of this

13

approach. Multiple measures, such as Pearson correlation and vector cosine based

similarity are used for this. [4] [6]

The user based top-N recommendation algorithm uses a similarity-based vector

model to identify the k most similar users to an active user. After the k most similar

users are found, their corresponding user-item matrices are aggregated to identify

the set of items to be recommended. A popular method to find the similar users is the

Locality-sensitive hashing, which implements the nearest neighbor mechanism in

linear time. [4] [6]

The advantages with this approach include: the explainability of the results, which is

an important aspect of recommendation systems; easy creation and use; easy

facilitation of new data; content-independence of the items being recommended;

good scaling with co-rated items. [4] [6]

There are also several disadvantages with this approach. Its performance decreases

when data gets sparse, which occurs frequently with web-related items. This interfere

the scalability of this approach and creates problems with large datasets. Although it

can efficiently handle new users because it relies on a data structure, adding new

items becomes more complicated since that representation usually relies on a

specific vector space. Adding new items requires inclusion of the new item and the

re-insertion of all the elements in the structure. [4] [6]

3.2.3 Model-based CF

Models are developed using data mining, machine learning algorithms to find

patterns based on training data. These are used to make predictions for real data.

There are many model-based CF algorithms. These include Bayesian networks,

clustering models, latent semantic models such as singular value decomposition,

probabilistic latent semantic analysis, Multiple Multiplicative Factor, Latent Dirichlet

allocation and Markov decision process based models. This approach has a more

holistic goal to uncover latent factors that explain observed ratings. Most of the

models are based on creating a classification or clustering technique to identify the

user based on the test set. The number of the parameters can be reduced based on

types of principal component analysis. There are several advantages with this

paradigm. It handles the data distribution better than memory based ones. This helps

14

with scalability with large data sets. It improves the prediction performance. It gives

an intuitive rationale for the recommendations. The disadvantages with this approach

are in the expensive model building. One needs to have a tradeoff between

prediction performance and scalability. One can lose useful information due to

reduction models. A number of models have difficulty explaining the predictions. [4]

[6]

3.2.4 k-Nearest Neighbors algorithm

The k-NN algorithm is one of the most popular among collaborative filtering (CF)

recommenders. This classification method – as most classifiers and clustering

techniques – is highly dependent on defining an appropriate similarity or distance

measure.

In pattern recognition, the k-Nearest Neighbors algorithm (or k-NN for short) is a non-

parametric method used for classification and regression.[1] In both cases, the input

consists of the k closest training examples in the feature space. The output depends

on whether k-NN is used for classification or regression: [4] [6]

In k-NN classification, the output is a class membership. An object is classified by

a majority vote of its neighbors, with the object being assigned to the class most

common among its k nearest neighbors (k is a positive integer, typically small). If

k = 1, then the object is simply assigned to the class of that single nearest

neighbor. [4]

In k-NN regression, the output is the property value for the object. This value is

the average of the values of its k nearest neighbors. [4]

k-NN is a type of instance-based learning, or lazy learning, where the function is only

approximated locally and all computation is deferred until classification. The k-NN

algorithm is among the simplest of all machine learning algorithms. [4]

Both for classification and regression, it can be useful to weight the contributions of

the neighbors, so that the nearer neighbors contribute more to the average than the

more distant ones. For example, a common weighting scheme consists in giving

each neighbor a weight of 1/d, where d is the distance to the neighbor. [4][6]

15

The neighbors are taken from a set of objects for which the class (for k-NN

classification) or the object property value (for k-NN regression) is known. This can

be thought of as the training set for the algorithm, though no explicit training step is

required. [4][6]

16

4. Impacts of Recommender Systems on Business and Users

After a short overview of basics of recommenders systems in the previous sections, it

is important to slowly go in the direction to describe what we have defined as

impacts. Based on different research papers, the impact of Recommender Systems

on the Business and on the User have often been neglected or understudied.

Therefore, the purpose of this thesis is to acquire deeper knowledge about such

impacts.

In order to analyze those impacts, the first important step is to define whether the

evaluation is done in Offline or Online experiments [17].

4.1 Offline and Online Evaluation

4.1.1 Offline Evaluation:

If an experiment is performed offline, the data is collected in advance and contains

the data set of user that have chosen or rated items. Offline Experiments are typically

the easiest ones to conduct. The main advantage is that no real interaction with a

user is required and allow us to test a wide range of recommender algorithms at a

low level of cost. A major disadvantage of offline evaluation is the fact that only a

small set of questions can be answered. Usually, it is the prediction power of an

algorithm. In offline evaluation, the behavior of the user is not modeled and must be

assumed. The influence of the Recommender System on the user’s behavior cannot

be measured in a direct way. As a consequence, it is not appropriate to only rely on

offline evaluations [6].

4.1.2 Online Evaluation:

The aim of online evaluation is to influence the user’s behavior. Online evaluation

tries to measure the change of the behavior in real time when the user has an

interaction with the Recommender System. Factor of the user’s intent, the user’s

context and the way how recommendations are presented play an important role.

Online Evaluation has a unique status as the measurement is done directly. The

downside of online evaluations is that the comparison of recommender algorithms is

costly and not always possible [6].

17

4.2 Accuracy

Accuracy is one the most important properties among the quality of Recommender

Systems. It decides about how the user’s information needs are fulfilled. Every user’s

personal information needs may vary lot as goals, preferences, knowledge and

contexts are on a very individual basis. While one user may be interested in the latest

music charts, another one’s goal might be to receive some recommendations about

Classical Music or Opera. Therefore, a recommender system has to be accurate by

recommending the most important items for the user [10].

4.3 Impacts of Recommender Systems on Business

In this paper, the focus lies on three types of impacts that Recommender Systems

can have on business. Figure 6 gives an overview of them:

Figure 6: Impacts of Recommender Systems on Business

Product Sales:

There are several reasons why Recommender Systems have an impact on the

Product Sales. A first reason is that the search costs for the user decrease as the

need to search for the right product is reduced. The amount of Product sales

increases as it is simpler for the user to find the product that satisfies his needs [11].

Product Views:

The Product Views of a User are the most important pre condition for the Product

Sales. However, the impact of Recommender Systems on Product Views is less

clear. A Recommender System can bring the user faster to his product with less

Product Sales

Product Views

Satisfaction of the Recommendation

Provider

Business

18

product views and clicks but RS have a positive effect on up-selling, cross-selling and

driving repeat visits which lead the product views to increase [11].

Satisfaction of the Recommendation Provider:

A good recommender system must not only satisfy the user but the Recommendation

Provider as well. One major interest of the recommendation provider is to operate

and maintain the recommender system at low costs. Costs may occur in labor,

memory, disk storage, CPU power and traffic [10]. Due to lack of evidence in

previous research papers, the Satisfaction of the Recommendation Provider will not

be taken into account in this thesis.

4.4 Impacts of Recommender Systems on Users

From the user's perspective, we have enumerated four main factors that influence

the user behavior (Figure 7).

Figure 7: Impacts of Recommender Systems on Users

The User satisfaction is a main goal of Recommender Systems. It’s important to

keep in mind that Accuracy alone does not necessarily contribute to User satisfaction

and other factors have to be included, i.e. serendipity. [10].

As an example, a recommendation from a grocery store to buy milk is indeed

accurate but it will not lead to satisfaction as it’s evident to everyone that milk is a

standard good. A recommendation for some specific dairy products like cheese or ice

User Satisaction

Online Time

User Preferences

User Perceptions

Users

19

cream would satisfy the customer more. User satisfaction can be influenced by many

more factors. Some of those are for example demographics, time spent till the

reception of recommendations or if costs are charged for the use of Recommender

Systems [10].

Online Time:

The Online time defines the time the user is spending on the Web page where the

Recommender System is running.

User Preferences:

This property is in a certain way difficult to control but interesting to analyze, given

the nature that it is easier for humans to give judgments than to give scores based on

their personal experiences, then we can rely on the system that had the largest

number of votes. However there are some concerns to consider in this property like it

assumes that all users are equal, which is not true in all the cases. For instance an e-

shop website, a client may prefer the opinion of users who buy many items or the

opinion of the users who buy only one single item. Therefore in this case we need to

weight the votes by the importance of the user and giving this weight may not be

easy. [6]

As a final remark in this property, when we want to rely on user preferences, we need

to compare specific properties in order to have an overview of the user. Therefore,

although user satisfaction is important to measure, splitting satisfaction into smaller

components is critical to understand the entire system and improve it [6].

20

User Perceptions:

“User perceptions refer to the process by which an individual selects, organizes, and

interprets stimuli into a meaningful and coherent picture of the world” [12.] User

perceptions can be divided in attributes for non-personally oriented perceptions and

for personally-oriented perceptions.(See figure 8)

Figure 8: Attributes for Consumer Perceptions [12]

Although user perception has been identified as a user impact, it is not part of the

analysis of this document because given its numerous attributes to describe it, it

remains unclear and it is still a an open research question from the side of

recommender systems and an intense subject of study on the side of consumer

research, and marketing.

Attributes for non-personally oriented

perceptions

• popular

• affordable

• exclusive

• unique

• manufactured

• luxurious

• uncommon

• superior

Attributes for personally oriented perceptions

• exquisite

• leading

• influential

• successful

• well-regarded

• memorable

• attractive

21

5. Evaluation of Recommender Systems and their value for

Business and Users

As it was shown in the previous section, the impacts of recommender systems with

respect users and business have an important role in our seminar analysis,

nevertheless before to identify potential correlation and show some of our findings. It

is important to understand, evaluate and make clear the different properties that are

involved in the recommendation systems, in order to understand the trade-offs on

parameters to measure properties of recommenders which in future sections would

be linked in a certain way with the impacts.

5.1 Recommendation System Properties

Regardless the case of an online or offline analysis, it is important to know the

recommender properties in order to understand later their impacts on the business or

users. In the next subsections, we will give a precise and concise overview of them.

The figure 9 shows the most important properties.

Figure 9: Recommendation System Properties

It is fundamental to mention that some of the properties in a recommender system

can be trade-off, for instance giving less important to the accuracy rather than

diversity, risk, privacy, etc. and analyze the effect of this change on the overall

performance. As a consequence the combinations and the highly level of

customization make the task very complex and the proper way of gaining such

Recommendation

System Properties

User Prefence Prediction

Accuracy

Coverage

Confidence

Trust

Novelty

Serendipity

Diversity

Utility

Risk

Robustness

Privacy

Adaptivity

Scalability

22

understanding without intensive online testing or distinct to the opinions of domain

experts is still an open question. Nevertheless as we discuss further in the results

and also as part of our contribution, there is an strategy to find among the whole set

of scenarios, some relevant ones and reduce the complexity to find new interesting

scenarios to analyze.

In the next subsections, we provide a short explanation of these properties in order to

get a better understanding and see later their potential interconnection with respect

the impacts.

5.1.1 Prediction Accuracy

The property relies on a prediction engine which may predict user opinions over

items (e.g. ratings of movies) or probability of usage (e.g. shopping). The core

assumption is that the user will prefer a system that provides more accurate

predictions.

Prediction accuracy is normally independent of the user interface and it can be

measured in an offline experiment. This property in a study measures the accuracy

given a recommendation. It is important to emphasize that accuracy influence or

causes a user behavior with comparison to the case which has no recommendations

Accuracy for the purpose of our study can be viewed from three perspectives as we

mention in the figure 10 and we will describe them later when we will mention more

about the link of this property with the impacts mentioned before [6].

Figure 10: Accuracy Predictions

Accuracy

Ratings predictions

Usage predictions

Rankings of items

23

5.1.2 Coverage

In this seminar paper, the term coverage refers to the proportion of items or user

interactions from which the recommendation system can recommend.

As a consequence, coverage can have an impact on the accuracy property reviewed

before and deliver different results because of the delimitation of particular items or

user interactions therefore there is an important trade-off between coverage and

accuracy [6].

5.1.3 Confidence

Confidence in the recommendation context is defined as the system’s trust in its

recommendations or predictions. As we have described the algorithms in the

previous chapter for instance, collaborative filtering recommenders tend to improve

their accuracy as the amount of data over items grows. Similarly, the confidence in

the predicted property typically also grows with the amount of data. It is important to

mention that when the system shows confidence values in their recommendations,

the users can take a further step and make better decisions, for instance if a system

reports a low confidence in an item, then the user may tend to do further research on

the item before making a decision [6].

5.1.4 Trust

This property refers to the user’s trust in the system recommendation. For instance, it

may be beneficial for the system to recommend some few items that the user already

knows and likes with the purpose that the user observes that the system provides

reasonable recommendations, which may increase the trust in the system

recommendations for unknown items. In addition to this technique, another common

way of enhancing trust in the system is to explain the recommendations that the

system provides. [6]

24

5.1.5 Novelty

Novel recommendations are recommendations for items that the user did not know

about [6]. In applications that require novel recommendation, a practical approach is

to filter out items that the user already rated or used. However, in many cases users

will not report all the items they have used in the past. Thus, this simple method is

insufficient to filter out all items that the user already knows and a more clever way to

split the information should be implemented. Another method for evaluating novel

recommendations uses the above assumption that popular items are less likely to be

novel [6].

5.1.6 Serendipity

Serendipity is a measure of how surprising the successful recommendations are. For

instance, if the user has rated positively many songs where a certain singer appears,

recommending the new song of that singer may be novel, because the user may not

know of it, but is hardly surprising. In the opposite direction, random

recommendations may be very surprising also, and therefore the need of balance

between serendipity and accuracy is required [6].

5.1.7 Diversity

Diversity goes in the opposite direction of similarity. It states that in some cases

suggesting a set of similar items may not be as useful for the user, because it may

take longer to explore the range of items.

For example, take in consideration a recommendation for a vacation, where the

system should recommend vacation packages. Presenting a list with 10

recommendations, all for the same location, varying only on the choice of hotel, or

the selection of attraction, may not be as useful as suggesting 10 different locations.

The user can view the various recommended locations and request more details on a

subset of the locations that are appropriate to her/him [6].

5.1.8 Utility

In general perspective, we can define various types of utility functions that the

recommender tries to optimize. Utility in the context of recommendations can be

interpreted and integrated with other properties such as: diversity or serendipity.

Nevertheless in the way that we interpret utility in this seminar paper is with respect

the value that either the system or the user gains from a recommendation [6].

25

5.1.9 Risk

A recommendation in some context may be associated with a potential risk. For

instance, in the stock market when recommending stocks for purchase, users may

wish to be risk-averse, preferring stocks that have a lower expected growth, but also

a lower risk of collapsing [6].

On the other hand, users may be risk-seeking, preferring stocks that have a

potentially high, even if less likely, profit. In such cases we may wish to evaluate not

only the (expected) value generated from a recommendation, but also to minimize

the risk [6].

5.1.10 Robustness

Robustness is the stability of the recommendation in the presence of fake information

typically inserted on purpose in order to influence the recommendations [6].

Influencing the system to change the rating of an item may be profitable to an

interested party. For example, an owner of a hotel may wish to boost the rating for

their hotel. This can be done by injecting fake user profiles that rate the hotel

positively, or by injecting fake users that rate the competitors negatively.

Such attempts to influence the recommendation are typically called attacks. The level

of protection varies depending form one protocol to another. Nevertheless, we should

be aware that creating a system that is immune to any type of attack is unrealistic [6].

5.1.11 Privacy

It is important for most users that their preferences stay private, that is, that no third

party can use the recommendation system to learn something about the preferences

of a specific user. For instance, consider the case where a user who is interested in

the wonderful and yet rare art of growing Bahamian orchids, then the user has

bought a book titled “The Divorce Organizer and Planner”. The spouse of that user,

looking for a present, upon browsing the book “The Bahamian and Caribbean

Species (Cattleyas and Their Relatives)” may get a recommendation of the type

“people who bought this book also bought” for the divorce organizer, thus revealing

sensitive private information [6].

26

5.1.12 Adaptivity

Real recommendation systems may operate in a setting where the item collection

changes rapidly, or where trends in interest over items may shift. As a simple

example of such systems is the recommendation of news items or related stories in

online newspapers. In this scenario stories may be interesting only over a short

period of time, afterwards becoming outdated. For instance, when an unexpected

news event occurs, such as the tsunami disaster, people become interested in

articles that may not have been interesting otherwise, such as a relatively old article

explaining the tsunami phenomenon [6].

5.1.13 Scalability

This property relies in the ability to navigate in large collections of items without

slowed down in the searches, one of the goals of the designers of such systems is to

scale up to real data sets. As such, it is often the case that algorithms trade other

properties, such as accuracy or coverage, for providing rapid results even for huge

data sets consisting of millions of items [6].

27

5.2 Summary of the Recommendation System Properties

The next Figure 11 shows a summary of all the RS-Properties considered in the

analysis in Chapter 6.

Dimension Metric/Technique Type(s) Accuracy Ratings: Root Mean Square Error (RMSE), Normalized RMSE

(NRMSE), Mean Absolute Error (MAE), Normalized MAE (NMAE)

Ranking: Normalized Distance-based Performance Measure (NDPM), Spearman correlation, Kendall correlation, Normal- ized Discounted Cumulative Gain (NDCG) Classification: Precision, Recall, False Positive Rate, Specifity, F-Measure, Reciver Operating Characteristics (ROC)

Quantitative

Coverage Catalogue Coverage, Weighted Catalogue Coverage, Prediction Coverage, Weighted Prediction Coverage

Quantitative

Confidence Neighborhood-aware similarity model, Similarity indicators Qualitative/ Quantitative

Trust User studies Qualitative Novelty Comparing recommendation list and user profiles, Counting

popular items Qualitative/ Quantitative

Serendipity Comparing recommendation list and user profiles, ratability Qualitative/ Quantitative

Diversity Diversity Measure, Relative Diversity, Precision-Diversity Curve, Q-Statistics, Set theoretic difference of recommendation lists

Quantitative

Utility Profit based utility function, study user intention, user study Qualitative/ Quantitative

Risk Depending on application and user preference Qualitative Robustness Prediction shift, average hit ratio, average rank Quantitative Privacy Differential privacy, RMSE vs. Differential privacy curve Qualitative/

Quantitative Adaptivity User studies but generally changing rate Quantitative/ Qualitative Scalability Training time, recommendation throughput Quantitative

Figure 11: Recommendation System Properties

Although there are many properties and we looked for papers that could include most

of these RS properties, there were not too many papers that could analyze or give

relevant evidence of interpretation other than accuracy, diversity and coverage

5.3 Evaluation Metrics for Recommender Systems

Evaluation Metrics or quality measures are a fundamental part to make a statement

about the quality of recommendation and prediction algorithms. The benefits of

evaluation metrics are that different Recommender Systems can be tested on

performance, compared with each other and improved. This stage provides an

overview about the most commonly used quality measures [7].

The evaluation metrics can be classified into 6 main groups. The first group consists

prediction metrics which are primarily used to test accuracy. Those are the Mean

Absolute Error (MAE), Root of Mean Square Error (RMSE) and Normalized Mean

28

Average Error (NMAE). In the second group, one can find the set of

recommendation metrics. Precision, Recall and Receiver Operating Characteristic

fall into that group. The third group deals with rank recommendation metrics. There

are two metrics which are the half-life and the discounted cumulative gain. Group Nr.

4 contains the diversity and the novelty of the recommended items. A fifth group

gives information stability metrics. Here, the use of the Mean Absolute Shift (MAS)

is proposed. The sixth and last group makes the use of Reliability measures [7].

Due the variety of options regarding the metrics, only the metrics that will be used in

Chapter 6 of this thesis are going to be presented.

5.3.1 Accuracy Metrics

Uis defined as the set of RS users, �̂�𝑢𝑖 as the set of predicted user-item pairs (u,i), 𝑟𝑢𝑖

as the true Ratings. The true ratings 𝑟𝑢𝑖 are known as they are hidden in an offline

experiment [6].

Mean Absolute Error (MAE)

Root of Mean Square Error (RMSE)

Normalized Mean Absolute Error (NMAE): The NMAE is a version of the MAE that

has been normalized by the range of (𝑟𝑚𝑎𝑥 − 𝑟min)

5.3.2 Privacy Metrics

Although in the recommender systems literature, there are many metrics definitions

to measure privacy, we find convenient to measure privacy with the k-anonymity

metric which is almost a standard in the context of Swiss Government Privacy. The

metric describes to which extent private information of the user is public. The

parameters of k-anonymity define the degree of anonymity. In the simplest approach,

the higher the value k, the more anonymity, otherwise values like k =0 means that all

private information is published in the Recommender System [19].

29

5.3.3 Adaptivity Metrics

The Adaptivity Metrics describe how fast the System can acquire new information

and can adapt it in the algorithm. Although this metrics can be measured by different

types of metrics, the one considered in this document is considered as changing

rate.

5.3.4 Trust Metrics

The Trustworthiness of a Recommender System cannot be measured in a

quantitative way. However, an often used approach is to conduct qualitative user

studies by asking the users whether the recommendations were reasonable to them.

There is the possibility to check how frequently a recommender is applied by a user.

A higher usage frequency is an indicator for a user to trust more in a Recommender

System [18].

5.3.5 Confidence Metrics

If we consider an online scenario, the confidence of a recommendation can be

calculated by the observation of environmental variables. Those so called confidence

scores indicate how frequently a recommender is used an applied by the user. Some

authors propose a calculation of confidences scores by using a neighborhood-aware

similarity model. The model includes similarities by users and items to generate

recommendations. The most suitable recommendation is the one that maximizes the

similarity between a recommended item and similar item. The usage of Confidence

scores can be applied by using confidence intervals or the probability that a value

predicted to the user is true [18].

5.3.6 Novelty Metrics

The Novelty Metrics measures the difference between the items recommended to

and known by the user. However, Novelty metrics do not have a standard. Bobadilla

et al. suggest the following formula used by various authors [7].

𝑍𝑢 represents the set of n recommendations to the user u. sim (i,j) refers to the

item-item memory based Collaborative Filtering similarity measures.

30

6. Correlation between evaluation metrics and impacts

After an extensive analysis on different research papers which are related to

recommender systems, we realized that in most of the cases the researchers present

papers in three general categories:

Technical Research Papers: Creating new or proposing variants of

algorithms

Applied Short Research Papers: Applying an algorithm for a case with

purpose of testing

Comparison Applied Research Papers: Applying different algorithms and

compare results.

In most of the cases they compare the results in performance of the algorithms by

analyzing the metrics and in a certain way a short description of these metrics used

by some algorithms with the properties related to them. The categories of papers

studied for this seminar are presented in Figure 12.

Recommender

AlgorithmsCASES

Results

Metrics analysis

Recommender

PropertiesImpacts

TEST Relation Relation Relation

Technical Research Papers

Applied Short Research Papers

Comparison Applied Research Papers

Consequences

Relation

Figure 12: Scope of our impacts analysis

However, as we can observe in the figure 12, the relation between the metrics and

the impacts has not been analyzed or mentioned in the studies. We found that in

most of the cases the interpretation of the results of the algorithms applied on certain

cases have absences of this analysis.

In this section we present a potential series of cases where the connection between

evaluation metrics and impacts is potentially present. On the one hand, in most of the

cases we will support our scenarios with some research papers references and on

the other hand there are cases that we consider to have some impacts but there is no

literature that support these assumptions, therefore these cases will be a suggestion

as a potential future work to continue the research in this direction.

31

6.1 Documented Scenarios

6.1.1 Impact of Accuracy on Online Time

Recommender

AlgorithmsCASES

Metrics

MAE

RMSE

NMAE

TEST Relation Relation RelationRS Property

Accuracy

Impact User

Online Time

Consequences

Keep Interest

↑ Visits

Buy

Figure 13: The impact of Accuracy on Online Time

Scenario:

The scenario presented in Figure 13 shows the proposed correlation from the metrics

that support accuracy to the Online Time impact and potential consequences on the

user side.

Key Assumptions:

Users who shop online have a low level of patience or time tolerance in terms

of waiting to receive the information, so the switching rate between websites

that provide the same service is very high.

A user that could not only find accurate results or receive a result after certain

amount of time will change the e-commerce platform provider.

Analysis:

The time to retrieve results as a performance indicator in the RS algorithms has been

documented and measured in different papers and it plays a main role to determine

the efficiency as we can observe specially in the reference [13], where they compare

the time performance of different algorithms 1-NN, 80-NN and Eigentaste-algorithm

on a case that recommends funny jokes to the user. The performance response

times are normally measured in milliseconds.

On the other side for our assumption 2, we found evidence on reference [14] about

the tolerable waiting time of users for computer response and we extracted the most

relevant findings in the figure 14.

32

Study Findings/Recommendations

Miller(1968) Delay of 2 seconds is the limit before interference with short term memory occurs

Nielsen(1993,1995,1996) Delay of 0.1 second is perceived as instantaneous success

Delay of 1.0 second is the limit for users ‘flow of thought to stay uninterrupted

Delay 10 seconds is the limit for keeping users ‘attention focus on the dialogue

Scheneiderman(1984) Delay of 2 seconds is the limit where response to simple commands becomes unacceptable to users

Figure 14: Summary of user's tolerable waiting time for computer response

In order to understand this case consider an example where an user is looking for a

particular flight itinerary for vacations and it chooses an online retailer but if the user

does not get a recommendation on time, he would rarely stay on the same page and

change into another provider to look again for the same flight itinerary. Based on the

previous studies, we can infer that an algorithm that is designed to provide accuracy

in the results, it is not successful if it does not show the recommendation to the user

in the waiting tolerance interval of the user.

Based on the evidence of the papers and the diagram in the figure 13, the online time

as an impact has some consequences for the user and although we cannot predict

the exact user behavior, we propose three different potential consequences that the

user can take as a next step:

Consequences:

Buy: The user simply gets the product.

Keep Interest: He finds the product but he does not buy it and keep interested

on it.

Increase the Number of visits: Regardless the user buys or keeps interest on

the product that he searches. The accuracy of the algorithm ensures the

success of the recommendations and increases the number of visits to the

page.

33

6.1.2 Impacts of Accuracy, Privacy and Adaptivity on User Preferences

Recommender

AlgorithmsCASES

Metrics

MAE

RMSE

NMAE

K-Anonymity

Changing Rate

TEST Relation Relation RelationRS Properties

Accuracy

Privacy

Adaptivity

Impact User

User Preference

Consequences

↑ Trust

↑ Visits

↑ Loyalty

Figure 15: The impact of Accuracy, Privacy and Adaptivity on User Preferences

Scenario:

As we reviewed in the recommender systems properties section, a recommender

algorithm can consider different kind of properties that are related to certain metrics

in this case we focus on the scenario where the recommender system focus on the

RS properties such as accuracy, privacy and adaptability in order to have an impact

on User Preferences.

Key Assumptions:

Privacy is very important for the user and any kind of misuse of the information

has a negative impact on the service that the user is using.

High accurate predictions let the user to continue using the system and have a

positive impact on the service.

A high changing rate in the recommended products give the sense to the user

that the recommender system is giving him the last products of the market or

that the system is freshly updated in the products that the e-commerce

platform is offering.

Analysis:

In this case we have mentioned in previous chapters the metrics related to the

accuracy such as: MAE, RMSE and NMAE. Nevertheless for properties as privacy

and adaptability the metrics that we consider to measure these RS properties are k-

anonymity and changing rate respectively.

34

In addition changing rate metric is related to the adaptivity of the algorithm in the

sense to measure the freshness of the data or external data elements that it is

considering for providing the recommendation.

Firstly, the assumption regarding the privacy relies on the potential risks to divulge

personal Information from the users in the recommendations because there is the

direct risk that someone will get information that the user wished to keep private. For

instance as supportive argument that we found in reference [15], it was found that

revealing identity information could lead to identity theft. There are also indirect risks

of re-identification finding information about a user in one system that could identify

her in another system. These elements cause a negative impact on the user that

modifies his user preferences in a negative way.

Nevertheless it is not sufficient to say that privacy can totally affect the user

preferences in the way that the user will stop to use the service. As an example taken

form reference [15], there is an important case analysis that supports this idea and it

comes from the situation where in 2004, Amazon’s Canadian site suddenly

accidentally revealed the identities of thousands of people, who had anonymously

posted book reviews. It turned out that authors were praising their own books and

trashing other authors’ books. The New York Times reported that “many people say

Amazon’s pages have turned into what one writer called ’a rhetorical war,’ where

friends and family members are regularly corralled to write glowing reviews and each

negative one is scrutinized for the digital fingerprints of known enemies.” To increase

the credibility of some reviews”, after these events the writers still used Amazon

services.

Secondly, the accuracy and adaptivity are very good complement properties to the

privacy in order to make an impact on the user in the user preference side in the

sense that if the recommender still provides good accuracy in the recommendations

and the freshness of the information is highly appreciate it by the user then he might

just be careful to not provide all his private information and still use the services of

the site. [16]

Finally, Based on the relation between the RS properties described above and the

user preferences, we consider that there are three potential consequences due to the

change of user preferences shown in Figure 15.

35

Consequences:

Loyalty: The user is more devoted to the product, if he higher preference.

Trust: The higher the User preference is, the more the user will trust in the

Recommendation.

Visits: The number of visits increase with higher User preference as he turns

into a buyer or stays an observer on the webpage.

6.1.3 Impact of Accuracy on Product Views

Recommender

AlgorithmsCASES

Metrics

MAE

RMSE

NMAE

TEST Relation Relation RelationRS Property

Accuracy

Impact Business

Product Views

Consequences CrossSelling

↑ Revenue

Up-Selling

Figure 16: The impact of Accuracy on Product Views

Scenario:

The scenario shows the potential impact of Accuracy Metrics on Product Views of the

user. As mentioned before in part 4.3 under reference [11], the impact of RS on

Product Views is somewhat ambiguous.

Here we present a duality effect:

An accurate Recommender System will decrease the time the user spends on

the webpage and therefore the Product Views decline.

The decrease in time that the user spends online catches the user’s interest to

view additional and differentiated products which increases the Product Views.

Assumptions:

We assume that duality effect referring to product views has a positive impact

on business in both cases.

Analysis:

In a Research Paper case for two top movie retailers in North America, a study about

the Impact of 3 different Collaborative Filtering Algorithms on Product Views was

conducted (Figure 17).

36

Figure 17: Average Items viewed for 1 Control Group and 3 Treatment Groups

In the study, the Average Number of Items viewed for a Control Group, to whom no

CF-algorithm was applied, scored 6.5464. For a treatment group where Purchased-

Based CF (“People who purchased this item also purchased”) was used, the Average

Number of Items viewed increased to 8.1684 associated with a p-value of 0.026 ≤

0.05 Error-Likelihood (statistically significant). Therefore, we can conclude that in the

little case described, the use of a Purchase-based CF-algorithm has a positive impact

on the amount of products views [11]. However, the use of view based CF only

increase the product views by score of 0.2070 and was statistically not significant.

The recently viewed algorithm implies a decrease of the item viewed by 0.3358.

Consequences:

Revenue Increase: The use of a Purchased-Based CF-algorithm seems to

have a positive influence on the Product Views. The increase of Products

viewed by users may therefore imply more potential buyers of products.

However, the relationship between Viewers and Buyers was not investigated

in the Research Paper study.

Cross Selling: If the product views increase, the user is very like to view also

ancillary items to his preferred product and will possibly buy them.

Up Selling: An increase of Product views can awake the customer’s interest to

buy a more expensive version of a product, an upgrade or an add-on. Up

Selling will be then an additional Revenue for the Business.

37

6.2 Non-documented Scenarios

6.2.1 Impacts of Trust on Product Sales and Satisfaction

Recommender

AlgorithmsCASES

Metrics

Similarity

TEST Relation Relation RelationRS Properties

Trust

Impact Business

Product Sales

Consequences

↑Prestige↑ Revenue

Impact User

Satisfaction

Consequences

↑ LoyaltyRepurchase

Figure 18: The impact of Trust on Product Sales and Satisfaction

Scenario:

Trust is an important determinant for Recommender Systems. Personal information

from the user is needed to give him suitable Recommendation in return. The trust a

RS needs to have from the user can be divided in two forms. Firstly, substantial

information of the user is essential to be able to give good recommendations.

Secondly, the details of an automated Recommender Systems are fairly visible to the

user. It is often the case that the user does not know how the recommendation is

being done for him. To be able to accept a recommendation, the user has to trust that

the recommendation has enough accuracy. There are three types of trust violations a

user consider: [15]

Exposure: Access to personal user data

Bias: Recommendation of users are being manipulated

Sabotage: The reduction of recommendation accuracy on purpose.

Key Assumptions:

The more a user trusts in the Recommendations System, the higher the user’s

usage frequency will be [18].

Analysis:

Based on the previous explanation of trust, the suspicion is that the level of trust can

either negatively or positively affect Product Sales and the User Satisfaction.

38

Consequences:

Of Product Sales:

Revenue increase

Prestige

Of User Satisfaction:

Repurchase

Loyalty

6.2.2 Impacts of Novelty/Confidence on Product Sales and User Preferences

Recommender

AlgorithmsCASES

Metrics

Sim(Items)

P-Values

TEST Relation Relation RelationRS Properties

Novelty

Confidence

Impact Business

Product Sales

Consequences

Popularity↑ Revenue

Impact User

User Preferences

Consequences

ReturnRepurchase

Figure 19: The impact of Novelty on Product Sales and User Intention

Scenario:

As mentioned in section 5.1.5, Novelty Recommendations refer to items about which

the user has not yet had knowledge. From the Business perspective, Novelty can

lead to additional Sales as the user discovers a new product which he gets excited

about and might purchase it. From the customer’s point of view, a new product might

have a positive impact on his preferences. However, if Novelty is viewed in an

isolated way, the user might only view the product but not purchase it. To be more

sure that Novelty has an effect on the user, one must consider Confidence at the

same the time.

Key Assumptions:

If a user discovers a new product and is confident with the Recommender

System (based on previous purchases) at the same time, he is more likely to

really buy the product which is positive for the User Preferences.

Analysis:

Although there was no evidence found in respected research papers, we can

conclude that if Novelty and Confidence are applied at the same time, the impact is

positive on the Product Sales.

39

Consequences:

Of Product Sales:

Increase in Revenue: The customer decides to buy the product which gives

additional revenue to the company.

Popularity: The user considers relies on the recommender system.

Of User Preferences:

Repurchase: If a user purchased a product under the condition that Novelty

and Confidence are together at the same time, then he’s likely to do the same

in future for a different product.

Return: When Novelty and Confidence are high but the user has not bought a

product yet, he might come back in the future which increases Product Views.

40

7. Analysis and Results

After the revision of the scenarios in the previous section, we arise to the point to

dedicate a chapter to analyze in details our findings and provide some observations

which are related to the correlation cases.

It is important to mention that in the previous section the scenarios presented had the

possibility to be documented by evidence found in previous research papers or case

studies. Nevertheless, there are some cases that cannot be still documented for the

moment but they are interesting to be analyzed.

In this chapter we mention briefly the cases which cannot be documented yet but that

are interesting to study in order to find more evidence and continue expanding the

correlations. Moreover, we add short observations on the cases that could be

documented in a short section.

Finally at the end of this chapter, we will summarize the relations among the different

metrics, recommender systems properties and impacts that are considered in our

seminar study.

7.1 Documented Scenarios

As we could observe in the previous correlation cases, on the one hand there is

enough evidence from the side of recommender system studies that mention a

relation between RS properties and their metrics in a technical interpretation, but on

the other hand studies from the side of consumer research field have an extensive

experience treating the impacts on users and their consequences such as user

behavior. In addition marketing in electronic business have also documented the link

between the impacts and their consequences in the business, like increment of

revenue, prestige or number or visits. The link between these different fields is

already experiencing an intersection and finding correlations becomes more visible

as the correlation cases described in our previous section.

7.2 Non-Documented Scenarios

Although there are many different recommender system (RS) properties identified

and quoted by several book references, only very few of them are mentioned or

applied in research papers. For instance, RS Properties such as accuracy,

41

confidence are among the most quoted in research papers rather than serendipity or

risk. In the recommendation section, we talk in more details about some particular

non-documented scenarios that can add a significant contribution and complement

the results of this seminar paper.

7.3 The correlation results between RS Properties Results and Impacts

The intention of this section is to show in more details our findings and potential

applications from our results and report our research contribution. Nonetheless,

before discussing our findings, applications and contributions, we would like to take

some time to come back to the figure 20, which was also shown in section 6, in order

to get back into the context and see clearly the directions of our results and

contributions.

As we described previously the three types of research papers categories found in

the course of our seminar research were mainly:

Technical Research Papers

Applied Short Research Papers

Comparison Applied Research Papers

We do not mention again the definitions of these three main identified categories but

it is important to emphasize the scope and the limit of the research paper

categories.(see figure 20)

Recommender

AlgorithmsCASES

Results

Metrics analysis

Recommender

PropertiesImpacts

TEST Relation Relation Relation

Technical Research Papers

Applied Short Research Papers

Comparison Applied Research Papers

Consequences

Relation

Figure 20: Research Paper Categories in Recommender Systems Field

Among all the categories, the research papers have in common that they could not

reach a deeper analysis with respect of the interpretation of the metrics or RS

properties with respect users or the business, they just limited to understand and

42

evaluate trade-offs on metrics to measure properties of recommenders and their

effect on the overall performance.

Moreover, based on the researcher’s profiles in the papers and their style to present

their results, we observed that they cannot make a deeper analysis on the effects of

the RS properties or metrics because they are limited on their expertise in the fields

such as Consumer Research or Marketing in Electronic Business and vice versa.

This fact is totally understandable and we are not saying that their research papers

are useless or with lack of results. Most of the researchers in the field of

recommender systems are involved in the computer sciences field and they are

somehow blind to see the correct direction to understand the effects of their

algorithms on the impact on business or users.

Based on the lack of direction, computer sciences or recommender system designers

still believe that the proper way of gaining such understanding of the impacts on

consumers or business is trying all the possible scenarios, different algorithms and

see the effects on the user or business. It is like trying to break the enigma

cryptographic machine trying all the 158 million combinations.

Maybe we exaggerated a little bit with this example but considering the scenarios

previously analyzed, we consider that understanding the proven consequences and

impacts from the side of marketing and consumer research can simplify the number

of scenarios that must be tested in order to understand better the impacts of

recommender algorithms on the users and business.

The Figure 21 suggests the direction of the recommender system design from the

side of consumer research, marketing and computer science fields.

43

Recommender

AlgorithmsCASES

Results

Metrics analysis

Recommender

PropertiesImpacts

TEST Relation Relation Corrrelation

Technical Research Papers

Applied Short Research Papers

Comparison Applied Research Papers

Consequences

Relation

Consumer Research and E-Marketing field Computer Science on Recommerder Systems field

Testing Assumptions

Suggested Direction of the Recommender System Design

Figure 21: Direction of the Recommender System Design

It is time to present our results and their interpretation in order to justify our previous

arguments. Figure 22 summarizes all the documented scenarios in one diagram:

Relations between metrics, RS properties, impacts and consequences.

RelationRS Property

Accuracy

Impact User

Online TimeKeep

Interest↑ Visits Buy

Metrics

K-Anonymity

Changing Rate

RelationRS Properties

Accuracy

Privacy

Adaptivity

Impact User

User Preference↑ Trust ↑ Visits↑ Loyalty

Metrics

MAE

RMSE

NMAE

RelationRelationRS Property

Accuracy

Impact Business

Product Views

CrossSelling

↑ Revenue Up-Selling

RelationConsequences

Consequences

Consequences

Figure 22: Relations between Metrics, RS properties and Impacts on Documented Scenarios

The Figure 22 displays all the founded documented scenarios with their relations and

their direction from the left to the right. On the one hand, fields like Marketing and

Consumer Research document extensively the relation between the impacts and the

consequences. On the other hand, computer scientists have documented extensively

the relation between the metrics and the RS properties.

44

Based on our documented scenarios the results show the missing correlation that is

present between the RS Properties and the Impacts (red arrow, figure 21). It is

important to mention that the relation found goes only in one direction (unidirectional)

and it does not go in the opposite one. For instance let’s see the case of Accuracy

causes an impact on Online Time, in one direction (left to right) we documented

already that the accuracy influences the user to remain more time on the website but

in the opposite direction (right to left) the reason that the user stays online on a

website does not necessary mean that the website has accuracy as a property.

The diagram in figure 22 helps for instance to the RS designers to know the

consequences of the metrics implemented by their recommender algorithms without

knowing the details of the consumer research theory but knowing the final

consequence effects on the user or business.

Marketers or managers for instance can select some desired consequences and

know which recommender algorithm metrics have the desired effect on the users or

business and buy or rely on a recommender algorithm that uses these metrics.

Our results show that finding the missing correlation between the RS properties and

impacts bring great benefits and increase the understanding not only of the impacts

of the business but it also reduces the complexity for recommender systems

designers to focus only on the metrics that create the desired effect on the side of

business or users.

45

8. Recommendations

The aim of the section is to offer a series of recommendations based on the previous

analysis, observations and results of this seminar document, in order that

researchers can continue in the direction as suggested in this document.

8.1 Variety in Recommender System Properties

In the meanwhile of our research paper seminar, we found several papers focusing

mostly on few RS properties such as accuracy or confidence. Testing not only the

performance of different algorithms brings a great idea of their overall performance

but also researchers should start to integrate in their research analysis where they

include more RS properties to complement their research or case studies.

In order to facilitate the test and design of new algorithms, we present research

suggestions from the marketing and consumer research side as non-documented

cases that can be tested in order to see if there is a correlation between the RS

properties and the impacts. These scenarios are based in the most documented

topics between Impacts and consequences among users and business.

8.1.1 Non-documented scenarios suggestions

The figure 23 present the relation between all the elements related to the non

documented suggested scenarios.

Metrics

SimilarityRelation

RelationRS Properties

Trust

Impact Business

Product SalesConsequences

↑Prestige↑ Revenue

Impact User

Satisfaction Consequences↑ LoyaltyRepurchase

Metrics

Sim(Items)

P-Values

Relation

RelationRS Properties

Novelty

Confidence

Impact Business

Product SalesConsequences

Popularity↑ Revenue

Impact User

User PreferencesConsequences

ReturnRepurchase

Link

Link

Figure 23: Relations between Metrics, RS properties and Impacts on non-documented Scenarios

46

The previous suggested scenarios are open to be expanded and they were

suggested from the point of view of marketing and consumer research. The

importance here is to find potential evidence that there is a correlation between the

RS properties and the impacts.

8.2 Perception of the User

Perhaps perception of the user is one of the most documented impacts in the field of

marketing and consumer research but still without a full understanding of it. As we

mentioned before, perception impact was not considered in this seminar document

due the complexity and all the wide variety of factors that involve the perception of

the user.

Future Recommender Algorithms that attempt to measure this characteristic should

be very robust and consider many different types of correlations among user

characteristics that are still not clear or still under investigation in the consumer

research field. Despite of the complexity to model this impact, the last mille in terms

of research direction in recommender systems should go in this way because once

an algorithm can predict the perception of the user, an algorithm can influence deeply

the particular human reasoning of each individual. (This situation is sometimes

documented as psychological manipulation).

8.3 New Research Questions

Meanwhile, we were analyzing and finding the different relations between the RS

properties and the impacts, we wanted to present as a part of our results a certain

kind of model that would allow us to understand in general if the success of a

recommender algorithm in terms to make a positive impact on the user or business

rely in the number of RS properties considered. Unfortunately given the nature of

research papers analyzed, they only consider a few number of RS properties and a

generalization is still too early to imply or deduce.

With the purpose to encourage future researchers, who have different knowledge

backgrounds like Consumer Research, Marketing and Computer Science, they could

look if there exist a linear or exponential behavior that can model the success of the

recommender algorithm on the impacts based on the RS properties considered.

The figure 24 shows the proposed models and the potential relation to prove

between the number of RS properties and the level of impact on users and business.

47

Level of Impact on the User

# RS Properties Considered0

Algorithm 1

Algorithm 3

Algorithm 2

Level of Impact on the Business

# RS Properties Considered0

Algorithm 1

Algorithm 3

Algorithm 2

Figure 24: Relation between RS Properties and their impact on Users and Business

48

9. Conclusion

The field of recommender systems is highly active and in constant evolution. On the

one hand researchers from the computer side field have focused strongly their efforts

to improve or create new algorithms and classify some of the their properties, but on

the other hand Consumer Research and Marketers have target their efforts to

understand the impacts and the behavior of the consumer.

At the end the final goal of the extensive study of recommender systems on the field

of electronic business is mainly focus to obtain a win-win relation between the

provider of a service (Business) and the customer.

After our exhaustive paper research, we detected an opportunity to create a good

synergy between two fields that seemed to be focusing their research in different

directions. As we presented in our results, both fields start to overlap each other and

based on this circumstances, we could find enough evidence that allowed us to

extend the interpretation of their research paper results and enhance their research

findings.

Furthermore, based on our results recommender systems designers, marketers or

consumer researchers can find easily the relations among the metrics needed in

certain scenarios to cause an assertive effect on the business or in the users as well

as they can also look at the desired consequences in order to have a clearer goal of

which algorithms should be applied.

Our seminar paper research gives a first approach of finding these correlations and

we are aware that there are much more relations to discover but it depends

enormously of the variety of RS properties tested in the algorithms. As long as the

diversity of studies in different RS properties increases, there will be more

possibilities for researchers to find more relations, which would allow them to refocus

their research in a better direction and discover interesting scenarios rather than

applying all their algorithms to all the possible scenario combinations.

49

10. Future Work

Based on the analysis, recommendations and conclusions presented in this seminar

document, it is important to mention the next steps or some research directions

derived from this paper. The next points represent a starting reference point where

researches can contribute with better interpretation results and continue with the

research of this seminar document. The next list mentions some recommendation

points derived from our research as a future work.

Researchers should start to test more variety of RS Properties in their papers,

rather than focusing on the most quoted RS properties such as accuracy and

confidence.

Continue their research of testing algorithms in the direction of testing the non-

documented scenarios proposed.

Research in the direction to find if there exists a relation between the impact of

the users or business vs the number of RS properties considered in the

algorithm

The perception of the user should start to be modeled in a more precise way

as long as the algorithms can take in consideration more properties without

compromising the performance of the algorithm.

Researchers should include Qualitative Analysis in their research.

50

11. References

[1].- Adomavicius, Gediminas, et al. "Do recommender systems manipulate consumer preferences? A study of anchoring effects." Information Systems Research 24.4 (2013): pp. 956 – 975 [2].- Chaffey, Dave. E-Business and E-Commerce Management: Strategy, Implementation and Practice. Harlow: Pearson; 2009. [3].- Ekstrand Michael D., Riedl John T., Konstan Joseph A. Collaborative Filtering Recommender Systems. Foundations and Trends in Human–Computer Interaction. Vol. 4, No. 2 (2010) 81–173 [4].- Jannach, Dietmar, et al. Recommender systems: An introduction. Cambridge University Press, 2010. [5].- Kowalczyk, Wojtek, Zoltán Szlávik, and Martijn C. Schut. "The impact of recommender systems on item, user, and ranking--diversity." Agents and Data Mining Interaction. Springer Berlin Heidelberg, 2012. 261-287. [6].- Rokach, Lior, Bracha Shapira, and Paul B.Kantor. Recommender systems handbook. Vol. 1. New York: Springer, 2011. [7].- Bobadilla et al. Recommender systems survey, in: Knowledge-Based Systems, Vol. 46 (2013), pp. 109-132 [8].- Beel, Jorean et al. Research Paper Recommender System Evaluation: A Quantitative Literature Survey, Magdeburg [9].- Lathia, N., Hailes, S., and Capra, L., The effect of correlation coefficients on communities of recommenders. In SAC ’08: Proceedings of the 2008 ACM symposium on Applied computing, pp. 2000–2005, New York, NY, USA, 2008. [10].- Joeran Beel, Stefan Langer, Marcel Genzmehr, Bela Gipp, Corinna Breitinger, and Andreas Nürnberger. Research paper recommender system evaluation: A quantitative literature survey. In Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation, RepSys '13, pages 15{22, New York, NY, USA, 2013. ACM. [11]. – Lee, Dokyun, Hosanagar, Kartik. Impact of Recommender Systems on Sales Volume and Diversity, in: Thirty Fifth International Conference on Information Systems, Auckland 2014. [12]. – Bambauer-Sache, Silke. Consumer Research 2014; Handout „Part A: Basic knowledge in the field of consumer research“ [13]. – Goldberg, Ken et al. Eigentaste: A Constant Time Collaborative Filtering Algorithm, University of California, Berkeley, IEOR and EECS Departments, 2000. [14]. - Nah, F. A study on tolerable waiting time: How long are Web users willing to wait? Behaviour & Information Technology, forthcoming, University of Nebraska-Lincoln, 2004.

51

[15]. – Karat, Marie-Claire, Blom Jan O., Karat John. Designing personalized user experiences in eCommerce, Human-Computer Iteraction Series Vol. 5, Kluwer Academic Publishers, 2004. [16]. – Lam, Shyong K “Tony”, Frankowski Dan, Riedl John. Do You Trust Your Recommendations? An Exploration Of Security and Privacy Issues in Recommender Systems, GroupLens Research, Computer Science and Engineering, University of Minnesota, 2005. [17]. – Bellogín, Alejandro et al. Report on the Workshop on Reproducibility and Replication in Recommender Systems Evaluation (RepSys), Centrum Wiskunde & Informatica, The Netherlands, Gravity R&D, Hungary, Universidad Autonoma de Madrid, Spain, 2013. [18]. - Robillard, M.P., Maalej, W., Walker, R.J., Zimmermann, Th. (Eds.) Recommendation Systems in Software Engineering. Springer Berlin-Heidelberg, 2014. [19]. - Federal Data Protection and Information Commissioner (FDPIC). Explanatory notes on Big Data, http://www.edoeb.admin.ch/datenschutz/00683/01169/index.html?lang=en, accessed on: April 29, 2015.