Seminar in E-Business & Recommender Systems
University of Fribourg, Department of Informatics
Research Paper
The impact of Recommender Systems on Business and Customers in Electronic Markets.
STUDENT NAMES: José A. Mancera, Philipp Bosshard STUDENT NUMBERS: 10-801-207, 06-200-844 COURSE NAME: Electronic Business and Recommender Systems
DEPARTMENT: Department of Informatics
SUPERVISOR: ASSISTANT: DATE OF SUBMISSION:
Prof. Dr. Andreas Meier Luis Terán 05-03-2015
II
Table of contents
List of Figures ............................................................................................................. V
1. Introduction ............................................................................................................. 1
1.1 Problem statement ............................................................................................ 1
1.2 Research Objectives and Methodology ............................................................. 2
1.2.1 Research Questions ................................................................................... 2
1.2.2 Objectives and Output of the thesis ............................................................ 2
1.2.3 Research Methodology ............................................................................... 2
1.3 Timetable .......................................................................................................... 3
1.4 Addressees ....................................................................................................... 3
2. Recommender Systems ......................................................................................... 3
2.1 Overview on Recommender Systems ............................................................... 3
2.2 Recommender System Categories ................................................................... 5
2.2.1 Collaborative Recommendation.................................................................. 5
2.2.2 Content-Based Recommendation ............................................................... 5
2.2.3 Knowledge-Based Recommendation ......................................................... 6
2.2.4 Hybrid Recommendation ............................................................................ 7
2.3 Most popular recommendation techniques per category ................................... 8
3. Recommender System Algorithms ....................................................................... 10
3.1 Similarity Measures ......................................................................................... 10
3.2 Collaborative Filtering Algorithms.................................................................... 11
4. Impacts of Recommender Systems on Business and Users ................................ 16
4.1 Offline and Online Evaluation .......................................................................... 16
4.1.1 Offline Evaluation: .................................................................................... 16
4.1.2 Online Evaluation: .................................................................................... 16
4.2 Accuracy ......................................................................................................... 17
4.3 Impacts of Recommender Systems on Business ............................................ 17
4.4 Impacts of Recommender Systems on Users ................................................. 18
III
5. Evaluation of Recommender Systems and their value for Business and Users ... 21
5.1 Recommendation System Properties .............................................................. 21
5.1.1 Prediction Accuracy .................................................................................. 22
5.1.2 Coverage .................................................................................................. 23
5.1.3 Confidence ............................................................................................... 23
5.1.4 Trust ......................................................................................................... 23
5.1.5 Novelty ..................................................................................................... 24
5.1.6 Serendipity ............................................................................................... 24
5.1.7 Diversity .................................................................................................... 24
5.1.8 Utility ......................................................................................................... 24
5.1.9 Risk .......................................................................................................... 25
5.1.10 Robustness............................................................................................. 25
5.1.11 Privacy .................................................................................................... 25
5.1.12 Adaptivity ................................................................................................ 26
5.1.13 Scalability ............................................................................................... 26
5.2 Summary of the Recommendation System Properties .................................... 27
5.3 Evaluation Metrics for Recommender Systems .............................................. 27
5.3.1 Accuracy Metrics ...................................................................................... 28
5.3.2 Privacy Metrics ......................................................................................... 28
5.3.3 Adaptivity Metrics ..................................................................................... 29
5.3.4 Trust Metrics ............................................................................................. 29
5.3.5 Confidence Metrics ................................................................................... 29
5.3.6 Novelty Metrics ......................................................................................... 29
6. Correlation between evaluation metrics and impacts ........................................... 30
6.1 Documented Scenarios ................................................................................... 31
6.1.1 Impact of Accuracy on Online Time .......................................................... 31
6.1.2 Impacts of Accuracy, Privacy and Adaptivity on User Preferences .......... 33
6.1.3 Impact of Accuracy on Product Views ...................................................... 35
IV
6.2 Non-documented Scenarios ............................................................................ 37
6.2.1 Impacts of Trust on Product Sales and Satisfaction ................................. 37
6.2.2 Impacts of Novelty/Confidence on Product Sales and User Preferences . 38
7. Analysis and Results ............................................................................................ 40
7.1 Documented Scenarios ................................................................................... 40
7.2 Non-Documented Scenarios ........................................................................... 40
7.3 The correlation results between RS Properties Results and Impacts ............. 41
8. Recommendations ................................................................................................ 45
8.1 Variety in Recommender System Properties .................................................. 45
8.1.1 Non-documented scenarios suggestions .................................................. 45
8.2 Perception of the User .................................................................................... 46
8.3 New Research Questions ............................................................................... 46
9. Conclusion ............................................................................................................ 48
10. Future Work ........................................................................................................ 49
11. References ......................................................................................................... 50
V
List of Figures
Figure 1: Recommender System Categories .............................................................. 4
Figure 2: Collaborative Recommendation Techniques ............................................... 8
Figure 3: Content-Based Recommendation Techniques ............................................ 9
Figure 4: Knowledge-Based Recommendation Techniques ....................................... 9
Figure 5: Hybrid Recommendation Techniques: ......................................................... 9
Figure 6: Impacts of Recommender Systems on Business ...................................... 17
Figure 7: Impacts of Recommender Systems on Users ........................................... 18
Figure 8: Attributes for Consumer Perceptions [12] .................................................. 20
Figure 9: Recommendation System Properties ........................................................ 21
Figure 10: Accuracy Predictions ............................................................................... 22
Figure 11: Recommendation System Properties ...................................................... 27
Figure 12: Scope of our impacts analysis ................................................................. 30
Figure 13: The impact of Accuracy on Online Time .................................................. 31
Figure 14: Summary of user's tolerable waiting time for computer response ........... 32
Figure 15: The impact of Accuracy, Privacy and Adaptivity on User Preferences .... 33
Figure 16: The impact of Accuracy on Product Views .............................................. 35
Figure 17: Average Items viewed for 1 Control Group and 3 Treatment Groups ...... 36
Figure 18: The impact of Trust on Product Sales and Satisfaction ........................... 37
Figure 19: The impact of Novelty on Product Sales and User Intention ................... 38
Figure 20: Research Paper Categories in Recommender Systems Field ................. 41
Figure 21: Direction of the Recommender System Design ....................................... 43
Figure 22: Relations between Metrics, RS properties and Impacts on Documented
Scenarios .................................................................................................................. 43
Figure 23: Relations between Metrics, RS properties and Impacts on non-
documented Scenarios ............................................................................................. 45
Figure 24: Relation between RS Properties and their impact on Users and Business
................................................................................................................................. 47
1
1. Introduction
1.1 Problem statement
Overview
In this age full of data and big amounts of information, people use a variety of
strategies to make choices about what to buy, how to spend their leisure time, and
even what to eat. Recommender systems automate some of these strategies with the
goal of providing to the user affordable, personal, and high-quality recommendations
in order to help the user in the decision-making process. After all, how can one be
sure that the user is not being manipulated and what is the recommenders influence
power on the users and on the business?
Firstly, the objective of this seminar thesis is to determine the impacts that
Recommender Systems can have on Business and Users from the side of consumer
research and marketing. Secondly, documentation and analysis of different research
papers was carried on in order to identify the most important evaluation metrics for
Recommender Systems. Thirdly, the document shows the results or our analysis
based on the evidence found concerning the correlation between evaluation metrics
and impacts of RS.
Finally, recommendations and suggestions for future work regarding the impacts of
Recommender Systems on Business and Customers will be proposed.
As an important remark, recommender Systems have multiple areas of application
consequently the focus of this paper only lies on e-business and e-commerce
segments (e.g. Amazon, eBay and other kinds of e-shops) and the position of
Recommender Systems for other types of segments are not part of the scope of this
document.
2
1.2 Research Objectives and Methodology
1.2.1 Research Questions
The next group of questions is the guideline of our study, each of them is answered
in sequence during the evolution of the document.
1. What are the impacts of Recommender System on users?
2. What are the impacts of Recommender System on business?
3. What are the most important evaluation metrics that can be used to analyze
influences on the Users and the Business?
4. What is the correlation between the evaluation metrics of and the influence of
recommender systems on business and users
5. Which aspects have been neglected in previous research papers?
1.2.2 Objectives and Output of the thesis
Based on the research questions, the main objectives of this seminar research paper
analysis are to find potential correlations between impacts and recommender system
(RS) properties. In addition recommendations and suggested paths of research are
also suggested further in the document, in order that other researchers can continue
our work and find more evidence about the metrics, which it would be essential to
design, improve or interpret the output of recommender systems. Understanding
these correlations requires a profound understanding about what recommender
systems are?, How do they work? and what are the influences that they have on
online customers and business?
1.2.3 Research Methodology
In order to answer the questions of this research seminar paper, our analysis would
be based not only on data of previous studies in the field of Recommender Systems,
consumer research and marketing but also on selected handbooks or textbooks
recommended by our supervisors. All these documentation would guide us to
discover or identify the variables that play a main role in the impact on users and
business in the segments of e-business and e-commerce.
3
1.3 Timetable
02-27-2015 Acceptance of working title
03-06-2015 Submission of the proposal
March 2015 Continue literature Research and reading
Writing Chapter 1, 2 and 3
April 2015 Writing Chapter 4
Draft of the paper
Finishing the report
Revision and Correction
April 2015 Midterm Appointment
05-03-2015 Submission of the thesis report
05-08-2015 Presentation of the thesis report
1.4 Addressees
The target audience of this paper is primarily students in the fields of computer
science, e-commerce, marketing and professionals who are involved in the field of
Recommender Systems. The results of this seminar document should provide the
parties mentioned above not only valuable knowledge in order to better understand,
analyze and improve the quality of Recommender Systems, but also a better
understanding of the consequences of the algorithms on users and business.
2. Recommender Systems
In order to get a better understanding and interpretation of the findings presented in
this seminar paper, it is important to review some core concepts in the field of
recommender systems before moving to the analysis and results. In the next two
chapters, there is a briefly overview of recommender systems, characteristics and
description of the algorithms considered in our analysis.
2.1 Overview on Recommender Systems
The majority of the Internet users have experienced in their visits to online stores or
web pages that offer their services, certain kind of recommendations from other users
and suggestions after reviewing a product as “Customers who bought this Item also
bought,” or “Customers who read this book also read these”. These
recommendations are becoming more often in the context of e-Commerce.[6]
4
Among all the different definitions, a Recommender System (RS) is software and
techniques which provide suggestions that determine whichever articles should be
shown to a particular visitor. [6]
Every RS requires that the system knows something about every user and it must
maintain a user model or user profile which contains for instance the user
preferences and remember the activity of the user, in order to be able to predict the
articles that might be interesting for the user. The way the RS collects this information
depends on the particular Recommendation technique, user preferences can, for
instance, be acquired implicitly by monitoring user behavior but recommender system
might also explicitly ask the visitor about his or her preferences. Moreover, it is
important to collect additional information, like opinions and tastes of a large
community and not only individual approaches.[6]
The variety in the information that can be collected is very wide but the most
important is to know which information the system could exploit when it generates a
list of personalized recommendations. Figure 1 gives an overview of the four main
RS categories:
Figure 1: Recommender System Categories
Recommender System
Collaborative
Content-Based
Knowledge-Based
Hybrid
5
2.2 Recommender System Categories
The next subsections give a short walkthrough around these four different RS
categories in order to understand the characteristics and parameters that are
involved in each of them
2.2.1 Collaborative Recommendation
The core idea of this type of recommender system is that if users shared the same
interests in the past and for instance the users that viewed or purchased the same
books will also have similar interests or preferences in the future.
As an example considers the case when user A and user B have a purchase history
that is strongly similar and user A has recently bought a book that B has not yet seen,
the basic rationale is to propose this book also to B. Because this selection of
hopefully interesting books involves filtering the most promising ones from a large set
and because the users implicitly collaborate with one another, this technique is also
called collaborative filtering (CF). [6]
Techniques under this category are wide used in the context of e-Commerce, the
advantage of these techniques is that the recommender system does not need to
know what the commodity or product is about, its genre, or who create it. Based on
these conditions to propose commodities that are actually similar to those the user
liked in the past might be more effective. [6]
2.2.2 Content-Based Recommendation
The techniques related to this category are based on the availability of item
descriptions and a profile that assigns importance to these characteristics. For
instance, thinking in an online store, the possible characteristics of the products might
include the genre, the specific topic, or the author. Similar to item descriptions, user
profiles may also be automatically derived and “learned” either by analyzing user
behavior and feedback or by asking explicitly about interests and preferences. [6]
In the context of content-based recommendation, the techniques are focused on how
the systems automatically acquire and continuously improve user profiles and how do
they determine which items match, similar items or common interest among users. [6]
6
Content-based recommendation has two advantages:
1. It does not require large user groups to achieve reasonable recommendation
accuracy.[6]
2. New items can be immediately recommended once item attributes are
available. [6]
2.2.3 Knowledge-Based Recommendation
There are market segments where the products are highly sophisticated, for instance
the consumer electronics, which involves not only a large number of one time buyers
but also the customer, does not have all the knowledge to understand about these
technologies. [6]
These simple facts bring new problems such that the recommender system cannot
rely on the existence of a purchase history, build user profiles or propose products
that other users bought. In addition more detailed and structured content must be
considered as technical and quality features. [6]
The solution that brings an answer in this context is a system that exploits additional
and means–end knowledge to generate recommendations. In such knowledge-based
approaches, the recommender system typically makes use of additional, often
manually provided, information about both the current user and the available items.
[6]
As a simple example of techniques in this category are Constraint based
recommenders which for instance in the case of the digital camera domain, a
constraint-based system could use detailed knowledge about the features of the
cameras, such as resolution, weight, or price. In addition, explicit constraints may be
used to describe the context in which certain features are relevant for the customer,
such as, for example, that a high resolution camera is advantageous if the customer
is interested in printing large pictures. Moreover the system could ask the user about
the relative importance of features, such as whether resolution is more important than
weight in order to provide recommendations. Although there are many different
techniques in this category, the recommender system should be able to answer
questions as: [6]
What are the mechanisms that can be used to select and rank the items based
on the user’s characteristics? [6]
7
How do we acquire the user profile in domains in which no purchase history is
available, and how can we take the customer’s explicit preferences into
account? [6]
Which interaction patterns can be used in interactive recommender systems?
[6]
Finally, in which dimensions can we personalize the dialog to maximize the
precision of the user preferences? [6]
2.2.4 Hybrid Recommendation
The previous categories have shown the main philosophy behind each of
recommender systems main categories, nevertheless combining recommendation
techniques can offer a better solution and more precise recommendations to specific
problems and the advantages and disadvantages depends on the problem setting.
[6]
For instance, community knowledge exists and detailed information about the
individual items is available, a recommender system could be enhanced by
hybridizing collaborative or social filtering with content-based techniques. In
particular, such a design could be used to overcome the described increase of
problems with pure collaborative approaches and rely on content analysis for new
items or new users. Combining different approaches, the recommender systems
should be able to answer: [6]
Which techniques can be combined, and what are the prerequisites for a given
combination? [6]
Should proposals be calculated for two or more systems sequentially, or do
other hybridization designs exist? [6]
How should the results of different techniques be weighted and can they be
determined dynamically? [6]
Last but not least, the classification and understanding of the philosophy behind each
category shows a general overview and it is important to remember that these
techniques vary constantly. The next section shows a more detailed classification of
some of these particular techniques per category.
8
2.3 Most popular recommendation techniques per category
Although there are many recommendation techniques used currently in the market
and mentioning all of them, would be impossible. This report shows some of them
and as general overview in order to give a better picture to the reader about the
outlook of the recommenders systems. The next figures show some of the
techniques by category.
Figure 2: Collaborative Recommendation Techniques
Collaborative
User-based nearest
neighbor
Item-based nearest
neighbor
User–item ratings matrix
SVD-based recommend
ation
Content-based
Content representatio
n and content
similarity
Similarity-based
retrieval
Text classification
methods
9
Figure 3: Content-Based Recommendation Techniques
Figure 4: Knowledge-Based Recommendation Techniques
Figure 5: Hybrid Recommendation Techniques:
Knowledge-based
recommendation
Constraint-based
recommenders
Case-based recommend
ers
Hybrid recommendation approaches
Monolithic hybridization
design
Parallelized hybridization
design
Pipelined hybridization
design
10
3. Recommender System Algorithms
It is essential to mention that the number of recommender systems that are available
for applications or in development are uncountable and it would be very extensive to
treat all of them in a document, in this section for the purposes of our research, we
have selected some of them which would be supportive later to make a deeper
analysis of some particular scenarios.
3.1 Similarity Measures
Similarity measures are normally applied as preprocessing data tools, which prepare
the data in order to apply a recommender algorithm. Similarity Measures are
considered as a complement of most of the algorithms. The next section shows the
most relevant ones that are used in most of the research papers that were
analyzed.[4]
3.1.1 Pearson Correlation
The similarity between items can also be given by their correlation which measures
the linear relationship between objects. Given the covariance of data points x and y
Σ, and their standard deviation σ, we compute the Pearson correlation using
[4]
3.1.2 Cosine
This measure indicates vector dot product and ||x|| ,which is the norm of vector x.
This similarity is known as the cosine similarity or the L2 Norm. Where σ is the
covariance matrix of the data. Another very common approach is to consider items as
document vectors of an n-dimensional space and compute their similarity as the
cosine (x) of the angle that they form: The cosine-based approach defines the
cosine-similarity between two users x and y as:[4]
11
3.1.3 Euclidean distance:
The simplest approach to measure similarity is the Euclidean distance, where d(x,y)
is the degree of the distance:
Where n is the number of dimensions (attributes) and 𝑥𝑘 and 𝑦𝑘 are the kth attributes
(components) of data objects x and y, respectively [6]
3.2 Collaborative Filtering Algorithms
During the seminar research, we found that most of the algorithms applied in different
research paper cases were mostly collaborative filtering algorithms. In this subsection
we mention the three main categories and the description of a particular algorithm
that was cited several times in different papers, k-Nearest Neighbors algorithm.
3.2.1 Item-based CF
Collaborative filtering (CF) is a technique used by some recommender systems.
Collaborative filtering has two senses, a narrow one and a more general one. In
general, collaborative filtering is the process of filtering for information or patterns
using techniques involving collaboration among multiple agents, viewpoints, data
sources, etc. Applications of collaborative filtering typically involve very large data
sets. Collaborative filtering methods have been applied to many different kinds of
data including: sensing and monitoring data, such as in mineral exploration,
environmental sensing over large areas or multiple sensors; financial data, such as
financial service institutions that integrate many financial sources; or in electronic
commerce and web applications where the focus is on user data, etc. The remainder
of this discussion focuses on collaborative filtering for user data, although some of
the methods and approaches may apply to the other major applications as well. [6]
In the newer, narrower sense, collaborative filtering is a method of making automatic
predictions (filtering) about the interests of a user by collecting preferences or taste
information from many users (collaborating). The underlying assumption of the
collaborative filtering approach is that if a person A has the same opinion as a person
B on an issue, A is more likely to have B's opinion on a different issue x than to have
12
the opinion on x of a person chosen randomly. For example, a collaborative filtering
recommendation system for television tastes could make predictions about which
television show a user should like given a partial list of that user's tastes (likes or
dislikes).Note that these predictions are specific to the user, but use information
gleaned from many users. This differs from the simpler approach of giving an
average (non-specific) score for each item of interest, for example based on its
number of votes. [4] [6]
3.2.2 Memory-based CF
This approach uses user rating data to compute the similarity between users or
items. This is used for making recommendations. This was an early approach used in
many commercial systems. It is effective and easy to implement. Typical examples of
this approach are neighborhood -based CF and item-based/user-based top-N
recommendations. For example, in user based approaches, the value of ratings user
'u' gives to item 'i' is calculated as an aggregation of some similar users' rating of the
item: [4] [6]
Where 'U' denotes the set of top 'N' users that are most similar to user 'u' who rated
item 'i'. Some examples of the aggregation function includes:
Where k is a normalizing factor defined as and is the average
rating of user u for all the items rated by u. [4] [6]
The neighborhood-based algorithm calculates the similarity between two users or
items, produces a prediction for the user by taking the weighted average of all the
ratings. Similarity computation between items or users is an important part of this
13
approach. Multiple measures, such as Pearson correlation and vector cosine based
similarity are used for this. [4] [6]
The user based top-N recommendation algorithm uses a similarity-based vector
model to identify the k most similar users to an active user. After the k most similar
users are found, their corresponding user-item matrices are aggregated to identify
the set of items to be recommended. A popular method to find the similar users is the
Locality-sensitive hashing, which implements the nearest neighbor mechanism in
linear time. [4] [6]
The advantages with this approach include: the explainability of the results, which is
an important aspect of recommendation systems; easy creation and use; easy
facilitation of new data; content-independence of the items being recommended;
good scaling with co-rated items. [4] [6]
There are also several disadvantages with this approach. Its performance decreases
when data gets sparse, which occurs frequently with web-related items. This interfere
the scalability of this approach and creates problems with large datasets. Although it
can efficiently handle new users because it relies on a data structure, adding new
items becomes more complicated since that representation usually relies on a
specific vector space. Adding new items requires inclusion of the new item and the
re-insertion of all the elements in the structure. [4] [6]
3.2.3 Model-based CF
Models are developed using data mining, machine learning algorithms to find
patterns based on training data. These are used to make predictions for real data.
There are many model-based CF algorithms. These include Bayesian networks,
clustering models, latent semantic models such as singular value decomposition,
probabilistic latent semantic analysis, Multiple Multiplicative Factor, Latent Dirichlet
allocation and Markov decision process based models. This approach has a more
holistic goal to uncover latent factors that explain observed ratings. Most of the
models are based on creating a classification or clustering technique to identify the
user based on the test set. The number of the parameters can be reduced based on
types of principal component analysis. There are several advantages with this
paradigm. It handles the data distribution better than memory based ones. This helps
14
with scalability with large data sets. It improves the prediction performance. It gives
an intuitive rationale for the recommendations. The disadvantages with this approach
are in the expensive model building. One needs to have a tradeoff between
prediction performance and scalability. One can lose useful information due to
reduction models. A number of models have difficulty explaining the predictions. [4]
[6]
3.2.4 k-Nearest Neighbors algorithm
The k-NN algorithm is one of the most popular among collaborative filtering (CF)
recommenders. This classification method – as most classifiers and clustering
techniques – is highly dependent on defining an appropriate similarity or distance
measure.
In pattern recognition, the k-Nearest Neighbors algorithm (or k-NN for short) is a non-
parametric method used for classification and regression.[1] In both cases, the input
consists of the k closest training examples in the feature space. The output depends
on whether k-NN is used for classification or regression: [4] [6]
In k-NN classification, the output is a class membership. An object is classified by
a majority vote of its neighbors, with the object being assigned to the class most
common among its k nearest neighbors (k is a positive integer, typically small). If
k = 1, then the object is simply assigned to the class of that single nearest
neighbor. [4]
In k-NN regression, the output is the property value for the object. This value is
the average of the values of its k nearest neighbors. [4]
k-NN is a type of instance-based learning, or lazy learning, where the function is only
approximated locally and all computation is deferred until classification. The k-NN
algorithm is among the simplest of all machine learning algorithms. [4]
Both for classification and regression, it can be useful to weight the contributions of
the neighbors, so that the nearer neighbors contribute more to the average than the
more distant ones. For example, a common weighting scheme consists in giving
each neighbor a weight of 1/d, where d is the distance to the neighbor. [4][6]
15
The neighbors are taken from a set of objects for which the class (for k-NN
classification) or the object property value (for k-NN regression) is known. This can
be thought of as the training set for the algorithm, though no explicit training step is
required. [4][6]
16
4. Impacts of Recommender Systems on Business and Users
After a short overview of basics of recommenders systems in the previous sections, it
is important to slowly go in the direction to describe what we have defined as
impacts. Based on different research papers, the impact of Recommender Systems
on the Business and on the User have often been neglected or understudied.
Therefore, the purpose of this thesis is to acquire deeper knowledge about such
impacts.
In order to analyze those impacts, the first important step is to define whether the
evaluation is done in Offline or Online experiments [17].
4.1 Offline and Online Evaluation
4.1.1 Offline Evaluation:
If an experiment is performed offline, the data is collected in advance and contains
the data set of user that have chosen or rated items. Offline Experiments are typically
the easiest ones to conduct. The main advantage is that no real interaction with a
user is required and allow us to test a wide range of recommender algorithms at a
low level of cost. A major disadvantage of offline evaluation is the fact that only a
small set of questions can be answered. Usually, it is the prediction power of an
algorithm. In offline evaluation, the behavior of the user is not modeled and must be
assumed. The influence of the Recommender System on the user’s behavior cannot
be measured in a direct way. As a consequence, it is not appropriate to only rely on
offline evaluations [6].
4.1.2 Online Evaluation:
The aim of online evaluation is to influence the user’s behavior. Online evaluation
tries to measure the change of the behavior in real time when the user has an
interaction with the Recommender System. Factor of the user’s intent, the user’s
context and the way how recommendations are presented play an important role.
Online Evaluation has a unique status as the measurement is done directly. The
downside of online evaluations is that the comparison of recommender algorithms is
costly and not always possible [6].
17
4.2 Accuracy
Accuracy is one the most important properties among the quality of Recommender
Systems. It decides about how the user’s information needs are fulfilled. Every user’s
personal information needs may vary lot as goals, preferences, knowledge and
contexts are on a very individual basis. While one user may be interested in the latest
music charts, another one’s goal might be to receive some recommendations about
Classical Music or Opera. Therefore, a recommender system has to be accurate by
recommending the most important items for the user [10].
4.3 Impacts of Recommender Systems on Business
In this paper, the focus lies on three types of impacts that Recommender Systems
can have on business. Figure 6 gives an overview of them:
Figure 6: Impacts of Recommender Systems on Business
Product Sales:
There are several reasons why Recommender Systems have an impact on the
Product Sales. A first reason is that the search costs for the user decrease as the
need to search for the right product is reduced. The amount of Product sales
increases as it is simpler for the user to find the product that satisfies his needs [11].
Product Views:
The Product Views of a User are the most important pre condition for the Product
Sales. However, the impact of Recommender Systems on Product Views is less
clear. A Recommender System can bring the user faster to his product with less
Product Sales
Product Views
Satisfaction of the Recommendation
Provider
Business
18
product views and clicks but RS have a positive effect on up-selling, cross-selling and
driving repeat visits which lead the product views to increase [11].
Satisfaction of the Recommendation Provider:
A good recommender system must not only satisfy the user but the Recommendation
Provider as well. One major interest of the recommendation provider is to operate
and maintain the recommender system at low costs. Costs may occur in labor,
memory, disk storage, CPU power and traffic [10]. Due to lack of evidence in
previous research papers, the Satisfaction of the Recommendation Provider will not
be taken into account in this thesis.
4.4 Impacts of Recommender Systems on Users
From the user's perspective, we have enumerated four main factors that influence
the user behavior (Figure 7).
Figure 7: Impacts of Recommender Systems on Users
The User satisfaction is a main goal of Recommender Systems. It’s important to
keep in mind that Accuracy alone does not necessarily contribute to User satisfaction
and other factors have to be included, i.e. serendipity. [10].
As an example, a recommendation from a grocery store to buy milk is indeed
accurate but it will not lead to satisfaction as it’s evident to everyone that milk is a
standard good. A recommendation for some specific dairy products like cheese or ice
User Satisaction
Online Time
User Preferences
User Perceptions
Users
19
cream would satisfy the customer more. User satisfaction can be influenced by many
more factors. Some of those are for example demographics, time spent till the
reception of recommendations or if costs are charged for the use of Recommender
Systems [10].
Online Time:
The Online time defines the time the user is spending on the Web page where the
Recommender System is running.
User Preferences:
This property is in a certain way difficult to control but interesting to analyze, given
the nature that it is easier for humans to give judgments than to give scores based on
their personal experiences, then we can rely on the system that had the largest
number of votes. However there are some concerns to consider in this property like it
assumes that all users are equal, which is not true in all the cases. For instance an e-
shop website, a client may prefer the opinion of users who buy many items or the
opinion of the users who buy only one single item. Therefore in this case we need to
weight the votes by the importance of the user and giving this weight may not be
easy. [6]
As a final remark in this property, when we want to rely on user preferences, we need
to compare specific properties in order to have an overview of the user. Therefore,
although user satisfaction is important to measure, splitting satisfaction into smaller
components is critical to understand the entire system and improve it [6].
20
User Perceptions:
“User perceptions refer to the process by which an individual selects, organizes, and
interprets stimuli into a meaningful and coherent picture of the world” [12.] User
perceptions can be divided in attributes for non-personally oriented perceptions and
for personally-oriented perceptions.(See figure 8)
Figure 8: Attributes for Consumer Perceptions [12]
Although user perception has been identified as a user impact, it is not part of the
analysis of this document because given its numerous attributes to describe it, it
remains unclear and it is still a an open research question from the side of
recommender systems and an intense subject of study on the side of consumer
research, and marketing.
Attributes for non-personally oriented
perceptions
• popular
• affordable
• exclusive
• unique
• manufactured
• luxurious
• uncommon
• superior
Attributes for personally oriented perceptions
• exquisite
• leading
• influential
• successful
• well-regarded
• memorable
• attractive
21
5. Evaluation of Recommender Systems and their value for
Business and Users
As it was shown in the previous section, the impacts of recommender systems with
respect users and business have an important role in our seminar analysis,
nevertheless before to identify potential correlation and show some of our findings. It
is important to understand, evaluate and make clear the different properties that are
involved in the recommendation systems, in order to understand the trade-offs on
parameters to measure properties of recommenders which in future sections would
be linked in a certain way with the impacts.
5.1 Recommendation System Properties
Regardless the case of an online or offline analysis, it is important to know the
recommender properties in order to understand later their impacts on the business or
users. In the next subsections, we will give a precise and concise overview of them.
The figure 9 shows the most important properties.
Figure 9: Recommendation System Properties
It is fundamental to mention that some of the properties in a recommender system
can be trade-off, for instance giving less important to the accuracy rather than
diversity, risk, privacy, etc. and analyze the effect of this change on the overall
performance. As a consequence the combinations and the highly level of
customization make the task very complex and the proper way of gaining such
Recommendation
System Properties
User Prefence Prediction
Accuracy
Coverage
Confidence
Trust
Novelty
Serendipity
Diversity
Utility
Risk
Robustness
Privacy
Adaptivity
Scalability
22
understanding without intensive online testing or distinct to the opinions of domain
experts is still an open question. Nevertheless as we discuss further in the results
and also as part of our contribution, there is an strategy to find among the whole set
of scenarios, some relevant ones and reduce the complexity to find new interesting
scenarios to analyze.
In the next subsections, we provide a short explanation of these properties in order to
get a better understanding and see later their potential interconnection with respect
the impacts.
5.1.1 Prediction Accuracy
The property relies on a prediction engine which may predict user opinions over
items (e.g. ratings of movies) or probability of usage (e.g. shopping). The core
assumption is that the user will prefer a system that provides more accurate
predictions.
Prediction accuracy is normally independent of the user interface and it can be
measured in an offline experiment. This property in a study measures the accuracy
given a recommendation. It is important to emphasize that accuracy influence or
causes a user behavior with comparison to the case which has no recommendations
Accuracy for the purpose of our study can be viewed from three perspectives as we
mention in the figure 10 and we will describe them later when we will mention more
about the link of this property with the impacts mentioned before [6].
Figure 10: Accuracy Predictions
Accuracy
Ratings predictions
Usage predictions
Rankings of items
23
5.1.2 Coverage
In this seminar paper, the term coverage refers to the proportion of items or user
interactions from which the recommendation system can recommend.
As a consequence, coverage can have an impact on the accuracy property reviewed
before and deliver different results because of the delimitation of particular items or
user interactions therefore there is an important trade-off between coverage and
accuracy [6].
5.1.3 Confidence
Confidence in the recommendation context is defined as the system’s trust in its
recommendations or predictions. As we have described the algorithms in the
previous chapter for instance, collaborative filtering recommenders tend to improve
their accuracy as the amount of data over items grows. Similarly, the confidence in
the predicted property typically also grows with the amount of data. It is important to
mention that when the system shows confidence values in their recommendations,
the users can take a further step and make better decisions, for instance if a system
reports a low confidence in an item, then the user may tend to do further research on
the item before making a decision [6].
5.1.4 Trust
This property refers to the user’s trust in the system recommendation. For instance, it
may be beneficial for the system to recommend some few items that the user already
knows and likes with the purpose that the user observes that the system provides
reasonable recommendations, which may increase the trust in the system
recommendations for unknown items. In addition to this technique, another common
way of enhancing trust in the system is to explain the recommendations that the
system provides. [6]
24
5.1.5 Novelty
Novel recommendations are recommendations for items that the user did not know
about [6]. In applications that require novel recommendation, a practical approach is
to filter out items that the user already rated or used. However, in many cases users
will not report all the items they have used in the past. Thus, this simple method is
insufficient to filter out all items that the user already knows and a more clever way to
split the information should be implemented. Another method for evaluating novel
recommendations uses the above assumption that popular items are less likely to be
novel [6].
5.1.6 Serendipity
Serendipity is a measure of how surprising the successful recommendations are. For
instance, if the user has rated positively many songs where a certain singer appears,
recommending the new song of that singer may be novel, because the user may not
know of it, but is hardly surprising. In the opposite direction, random
recommendations may be very surprising also, and therefore the need of balance
between serendipity and accuracy is required [6].
5.1.7 Diversity
Diversity goes in the opposite direction of similarity. It states that in some cases
suggesting a set of similar items may not be as useful for the user, because it may
take longer to explore the range of items.
For example, take in consideration a recommendation for a vacation, where the
system should recommend vacation packages. Presenting a list with 10
recommendations, all for the same location, varying only on the choice of hotel, or
the selection of attraction, may not be as useful as suggesting 10 different locations.
The user can view the various recommended locations and request more details on a
subset of the locations that are appropriate to her/him [6].
5.1.8 Utility
In general perspective, we can define various types of utility functions that the
recommender tries to optimize. Utility in the context of recommendations can be
interpreted and integrated with other properties such as: diversity or serendipity.
Nevertheless in the way that we interpret utility in this seminar paper is with respect
the value that either the system or the user gains from a recommendation [6].
25
5.1.9 Risk
A recommendation in some context may be associated with a potential risk. For
instance, in the stock market when recommending stocks for purchase, users may
wish to be risk-averse, preferring stocks that have a lower expected growth, but also
a lower risk of collapsing [6].
On the other hand, users may be risk-seeking, preferring stocks that have a
potentially high, even if less likely, profit. In such cases we may wish to evaluate not
only the (expected) value generated from a recommendation, but also to minimize
the risk [6].
5.1.10 Robustness
Robustness is the stability of the recommendation in the presence of fake information
typically inserted on purpose in order to influence the recommendations [6].
Influencing the system to change the rating of an item may be profitable to an
interested party. For example, an owner of a hotel may wish to boost the rating for
their hotel. This can be done by injecting fake user profiles that rate the hotel
positively, or by injecting fake users that rate the competitors negatively.
Such attempts to influence the recommendation are typically called attacks. The level
of protection varies depending form one protocol to another. Nevertheless, we should
be aware that creating a system that is immune to any type of attack is unrealistic [6].
5.1.11 Privacy
It is important for most users that their preferences stay private, that is, that no third
party can use the recommendation system to learn something about the preferences
of a specific user. For instance, consider the case where a user who is interested in
the wonderful and yet rare art of growing Bahamian orchids, then the user has
bought a book titled “The Divorce Organizer and Planner”. The spouse of that user,
looking for a present, upon browsing the book “The Bahamian and Caribbean
Species (Cattleyas and Their Relatives)” may get a recommendation of the type
“people who bought this book also bought” for the divorce organizer, thus revealing
sensitive private information [6].
26
5.1.12 Adaptivity
Real recommendation systems may operate in a setting where the item collection
changes rapidly, or where trends in interest over items may shift. As a simple
example of such systems is the recommendation of news items or related stories in
online newspapers. In this scenario stories may be interesting only over a short
period of time, afterwards becoming outdated. For instance, when an unexpected
news event occurs, such as the tsunami disaster, people become interested in
articles that may not have been interesting otherwise, such as a relatively old article
explaining the tsunami phenomenon [6].
5.1.13 Scalability
This property relies in the ability to navigate in large collections of items without
slowed down in the searches, one of the goals of the designers of such systems is to
scale up to real data sets. As such, it is often the case that algorithms trade other
properties, such as accuracy or coverage, for providing rapid results even for huge
data sets consisting of millions of items [6].
27
5.2 Summary of the Recommendation System Properties
The next Figure 11 shows a summary of all the RS-Properties considered in the
analysis in Chapter 6.
Dimension Metric/Technique Type(s) Accuracy Ratings: Root Mean Square Error (RMSE), Normalized RMSE
(NRMSE), Mean Absolute Error (MAE), Normalized MAE (NMAE)
Ranking: Normalized Distance-based Performance Measure (NDPM), Spearman correlation, Kendall correlation, Normal- ized Discounted Cumulative Gain (NDCG) Classification: Precision, Recall, False Positive Rate, Specifity, F-Measure, Reciver Operating Characteristics (ROC)
Quantitative
Coverage Catalogue Coverage, Weighted Catalogue Coverage, Prediction Coverage, Weighted Prediction Coverage
Quantitative
Confidence Neighborhood-aware similarity model, Similarity indicators Qualitative/ Quantitative
Trust User studies Qualitative Novelty Comparing recommendation list and user profiles, Counting
popular items Qualitative/ Quantitative
Serendipity Comparing recommendation list and user profiles, ratability Qualitative/ Quantitative
Diversity Diversity Measure, Relative Diversity, Precision-Diversity Curve, Q-Statistics, Set theoretic difference of recommendation lists
Quantitative
Utility Profit based utility function, study user intention, user study Qualitative/ Quantitative
Risk Depending on application and user preference Qualitative Robustness Prediction shift, average hit ratio, average rank Quantitative Privacy Differential privacy, RMSE vs. Differential privacy curve Qualitative/
Quantitative Adaptivity User studies but generally changing rate Quantitative/ Qualitative Scalability Training time, recommendation throughput Quantitative
Figure 11: Recommendation System Properties
Although there are many properties and we looked for papers that could include most
of these RS properties, there were not too many papers that could analyze or give
relevant evidence of interpretation other than accuracy, diversity and coverage
5.3 Evaluation Metrics for Recommender Systems
Evaluation Metrics or quality measures are a fundamental part to make a statement
about the quality of recommendation and prediction algorithms. The benefits of
evaluation metrics are that different Recommender Systems can be tested on
performance, compared with each other and improved. This stage provides an
overview about the most commonly used quality measures [7].
The evaluation metrics can be classified into 6 main groups. The first group consists
prediction metrics which are primarily used to test accuracy. Those are the Mean
Absolute Error (MAE), Root of Mean Square Error (RMSE) and Normalized Mean
28
Average Error (NMAE). In the second group, one can find the set of
recommendation metrics. Precision, Recall and Receiver Operating Characteristic
fall into that group. The third group deals with rank recommendation metrics. There
are two metrics which are the half-life and the discounted cumulative gain. Group Nr.
4 contains the diversity and the novelty of the recommended items. A fifth group
gives information stability metrics. Here, the use of the Mean Absolute Shift (MAS)
is proposed. The sixth and last group makes the use of Reliability measures [7].
Due the variety of options regarding the metrics, only the metrics that will be used in
Chapter 6 of this thesis are going to be presented.
5.3.1 Accuracy Metrics
Uis defined as the set of RS users, �̂�𝑢𝑖 as the set of predicted user-item pairs (u,i), 𝑟𝑢𝑖
as the true Ratings. The true ratings 𝑟𝑢𝑖 are known as they are hidden in an offline
experiment [6].
Mean Absolute Error (MAE)
Root of Mean Square Error (RMSE)
Normalized Mean Absolute Error (NMAE): The NMAE is a version of the MAE that
has been normalized by the range of (𝑟𝑚𝑎𝑥 − 𝑟min)
5.3.2 Privacy Metrics
Although in the recommender systems literature, there are many metrics definitions
to measure privacy, we find convenient to measure privacy with the k-anonymity
metric which is almost a standard in the context of Swiss Government Privacy. The
metric describes to which extent private information of the user is public. The
parameters of k-anonymity define the degree of anonymity. In the simplest approach,
the higher the value k, the more anonymity, otherwise values like k =0 means that all
private information is published in the Recommender System [19].
29
5.3.3 Adaptivity Metrics
The Adaptivity Metrics describe how fast the System can acquire new information
and can adapt it in the algorithm. Although this metrics can be measured by different
types of metrics, the one considered in this document is considered as changing
rate.
5.3.4 Trust Metrics
The Trustworthiness of a Recommender System cannot be measured in a
quantitative way. However, an often used approach is to conduct qualitative user
studies by asking the users whether the recommendations were reasonable to them.
There is the possibility to check how frequently a recommender is applied by a user.
A higher usage frequency is an indicator for a user to trust more in a Recommender
System [18].
5.3.5 Confidence Metrics
If we consider an online scenario, the confidence of a recommendation can be
calculated by the observation of environmental variables. Those so called confidence
scores indicate how frequently a recommender is used an applied by the user. Some
authors propose a calculation of confidences scores by using a neighborhood-aware
similarity model. The model includes similarities by users and items to generate
recommendations. The most suitable recommendation is the one that maximizes the
similarity between a recommended item and similar item. The usage of Confidence
scores can be applied by using confidence intervals or the probability that a value
predicted to the user is true [18].
5.3.6 Novelty Metrics
The Novelty Metrics measures the difference between the items recommended to
and known by the user. However, Novelty metrics do not have a standard. Bobadilla
et al. suggest the following formula used by various authors [7].
𝑍𝑢 represents the set of n recommendations to the user u. sim (i,j) refers to the
item-item memory based Collaborative Filtering similarity measures.
30
6. Correlation between evaluation metrics and impacts
After an extensive analysis on different research papers which are related to
recommender systems, we realized that in most of the cases the researchers present
papers in three general categories:
Technical Research Papers: Creating new or proposing variants of
algorithms
Applied Short Research Papers: Applying an algorithm for a case with
purpose of testing
Comparison Applied Research Papers: Applying different algorithms and
compare results.
In most of the cases they compare the results in performance of the algorithms by
analyzing the metrics and in a certain way a short description of these metrics used
by some algorithms with the properties related to them. The categories of papers
studied for this seminar are presented in Figure 12.
Recommender
AlgorithmsCASES
Results
Metrics analysis
Recommender
PropertiesImpacts
TEST Relation Relation Relation
Technical Research Papers
Applied Short Research Papers
Comparison Applied Research Papers
Consequences
Relation
Figure 12: Scope of our impacts analysis
However, as we can observe in the figure 12, the relation between the metrics and
the impacts has not been analyzed or mentioned in the studies. We found that in
most of the cases the interpretation of the results of the algorithms applied on certain
cases have absences of this analysis.
In this section we present a potential series of cases where the connection between
evaluation metrics and impacts is potentially present. On the one hand, in most of the
cases we will support our scenarios with some research papers references and on
the other hand there are cases that we consider to have some impacts but there is no
literature that support these assumptions, therefore these cases will be a suggestion
as a potential future work to continue the research in this direction.
31
6.1 Documented Scenarios
6.1.1 Impact of Accuracy on Online Time
Recommender
AlgorithmsCASES
Metrics
MAE
RMSE
NMAE
TEST Relation Relation RelationRS Property
Accuracy
Impact User
Online Time
Consequences
Keep Interest
↑ Visits
Buy
Figure 13: The impact of Accuracy on Online Time
Scenario:
The scenario presented in Figure 13 shows the proposed correlation from the metrics
that support accuracy to the Online Time impact and potential consequences on the
user side.
Key Assumptions:
Users who shop online have a low level of patience or time tolerance in terms
of waiting to receive the information, so the switching rate between websites
that provide the same service is very high.
A user that could not only find accurate results or receive a result after certain
amount of time will change the e-commerce platform provider.
Analysis:
The time to retrieve results as a performance indicator in the RS algorithms has been
documented and measured in different papers and it plays a main role to determine
the efficiency as we can observe specially in the reference [13], where they compare
the time performance of different algorithms 1-NN, 80-NN and Eigentaste-algorithm
on a case that recommends funny jokes to the user. The performance response
times are normally measured in milliseconds.
On the other side for our assumption 2, we found evidence on reference [14] about
the tolerable waiting time of users for computer response and we extracted the most
relevant findings in the figure 14.
32
Study Findings/Recommendations
Miller(1968) Delay of 2 seconds is the limit before interference with short term memory occurs
Nielsen(1993,1995,1996) Delay of 0.1 second is perceived as instantaneous success
Delay of 1.0 second is the limit for users ‘flow of thought to stay uninterrupted
Delay 10 seconds is the limit for keeping users ‘attention focus on the dialogue
Scheneiderman(1984) Delay of 2 seconds is the limit where response to simple commands becomes unacceptable to users
Figure 14: Summary of user's tolerable waiting time for computer response
In order to understand this case consider an example where an user is looking for a
particular flight itinerary for vacations and it chooses an online retailer but if the user
does not get a recommendation on time, he would rarely stay on the same page and
change into another provider to look again for the same flight itinerary. Based on the
previous studies, we can infer that an algorithm that is designed to provide accuracy
in the results, it is not successful if it does not show the recommendation to the user
in the waiting tolerance interval of the user.
Based on the evidence of the papers and the diagram in the figure 13, the online time
as an impact has some consequences for the user and although we cannot predict
the exact user behavior, we propose three different potential consequences that the
user can take as a next step:
Consequences:
Buy: The user simply gets the product.
Keep Interest: He finds the product but he does not buy it and keep interested
on it.
Increase the Number of visits: Regardless the user buys or keeps interest on
the product that he searches. The accuracy of the algorithm ensures the
success of the recommendations and increases the number of visits to the
page.
33
6.1.2 Impacts of Accuracy, Privacy and Adaptivity on User Preferences
Recommender
AlgorithmsCASES
Metrics
MAE
RMSE
NMAE
K-Anonymity
Changing Rate
TEST Relation Relation RelationRS Properties
Accuracy
Privacy
Adaptivity
Impact User
User Preference
Consequences
↑ Trust
↑ Visits
↑ Loyalty
Figure 15: The impact of Accuracy, Privacy and Adaptivity on User Preferences
Scenario:
As we reviewed in the recommender systems properties section, a recommender
algorithm can consider different kind of properties that are related to certain metrics
in this case we focus on the scenario where the recommender system focus on the
RS properties such as accuracy, privacy and adaptability in order to have an impact
on User Preferences.
Key Assumptions:
Privacy is very important for the user and any kind of misuse of the information
has a negative impact on the service that the user is using.
High accurate predictions let the user to continue using the system and have a
positive impact on the service.
A high changing rate in the recommended products give the sense to the user
that the recommender system is giving him the last products of the market or
that the system is freshly updated in the products that the e-commerce
platform is offering.
Analysis:
In this case we have mentioned in previous chapters the metrics related to the
accuracy such as: MAE, RMSE and NMAE. Nevertheless for properties as privacy
and adaptability the metrics that we consider to measure these RS properties are k-
anonymity and changing rate respectively.
34
In addition changing rate metric is related to the adaptivity of the algorithm in the
sense to measure the freshness of the data or external data elements that it is
considering for providing the recommendation.
Firstly, the assumption regarding the privacy relies on the potential risks to divulge
personal Information from the users in the recommendations because there is the
direct risk that someone will get information that the user wished to keep private. For
instance as supportive argument that we found in reference [15], it was found that
revealing identity information could lead to identity theft. There are also indirect risks
of re-identification finding information about a user in one system that could identify
her in another system. These elements cause a negative impact on the user that
modifies his user preferences in a negative way.
Nevertheless it is not sufficient to say that privacy can totally affect the user
preferences in the way that the user will stop to use the service. As an example taken
form reference [15], there is an important case analysis that supports this idea and it
comes from the situation where in 2004, Amazon’s Canadian site suddenly
accidentally revealed the identities of thousands of people, who had anonymously
posted book reviews. It turned out that authors were praising their own books and
trashing other authors’ books. The New York Times reported that “many people say
Amazon’s pages have turned into what one writer called ’a rhetorical war,’ where
friends and family members are regularly corralled to write glowing reviews and each
negative one is scrutinized for the digital fingerprints of known enemies.” To increase
the credibility of some reviews”, after these events the writers still used Amazon
services.
Secondly, the accuracy and adaptivity are very good complement properties to the
privacy in order to make an impact on the user in the user preference side in the
sense that if the recommender still provides good accuracy in the recommendations
and the freshness of the information is highly appreciate it by the user then he might
just be careful to not provide all his private information and still use the services of
the site. [16]
Finally, Based on the relation between the RS properties described above and the
user preferences, we consider that there are three potential consequences due to the
change of user preferences shown in Figure 15.
35
Consequences:
Loyalty: The user is more devoted to the product, if he higher preference.
Trust: The higher the User preference is, the more the user will trust in the
Recommendation.
Visits: The number of visits increase with higher User preference as he turns
into a buyer or stays an observer on the webpage.
6.1.3 Impact of Accuracy on Product Views
Recommender
AlgorithmsCASES
Metrics
MAE
RMSE
NMAE
TEST Relation Relation RelationRS Property
Accuracy
Impact Business
Product Views
Consequences CrossSelling
↑ Revenue
Up-Selling
Figure 16: The impact of Accuracy on Product Views
Scenario:
The scenario shows the potential impact of Accuracy Metrics on Product Views of the
user. As mentioned before in part 4.3 under reference [11], the impact of RS on
Product Views is somewhat ambiguous.
Here we present a duality effect:
An accurate Recommender System will decrease the time the user spends on
the webpage and therefore the Product Views decline.
The decrease in time that the user spends online catches the user’s interest to
view additional and differentiated products which increases the Product Views.
Assumptions:
We assume that duality effect referring to product views has a positive impact
on business in both cases.
Analysis:
In a Research Paper case for two top movie retailers in North America, a study about
the Impact of 3 different Collaborative Filtering Algorithms on Product Views was
conducted (Figure 17).
36
Figure 17: Average Items viewed for 1 Control Group and 3 Treatment Groups
In the study, the Average Number of Items viewed for a Control Group, to whom no
CF-algorithm was applied, scored 6.5464. For a treatment group where Purchased-
Based CF (“People who purchased this item also purchased”) was used, the Average
Number of Items viewed increased to 8.1684 associated with a p-value of 0.026 ≤
0.05 Error-Likelihood (statistically significant). Therefore, we can conclude that in the
little case described, the use of a Purchase-based CF-algorithm has a positive impact
on the amount of products views [11]. However, the use of view based CF only
increase the product views by score of 0.2070 and was statistically not significant.
The recently viewed algorithm implies a decrease of the item viewed by 0.3358.
Consequences:
Revenue Increase: The use of a Purchased-Based CF-algorithm seems to
have a positive influence on the Product Views. The increase of Products
viewed by users may therefore imply more potential buyers of products.
However, the relationship between Viewers and Buyers was not investigated
in the Research Paper study.
Cross Selling: If the product views increase, the user is very like to view also
ancillary items to his preferred product and will possibly buy them.
Up Selling: An increase of Product views can awake the customer’s interest to
buy a more expensive version of a product, an upgrade or an add-on. Up
Selling will be then an additional Revenue for the Business.
37
6.2 Non-documented Scenarios
6.2.1 Impacts of Trust on Product Sales and Satisfaction
Recommender
AlgorithmsCASES
Metrics
Similarity
TEST Relation Relation RelationRS Properties
Trust
Impact Business
Product Sales
Consequences
↑Prestige↑ Revenue
Impact User
Satisfaction
Consequences
↑ LoyaltyRepurchase
Figure 18: The impact of Trust on Product Sales and Satisfaction
Scenario:
Trust is an important determinant for Recommender Systems. Personal information
from the user is needed to give him suitable Recommendation in return. The trust a
RS needs to have from the user can be divided in two forms. Firstly, substantial
information of the user is essential to be able to give good recommendations.
Secondly, the details of an automated Recommender Systems are fairly visible to the
user. It is often the case that the user does not know how the recommendation is
being done for him. To be able to accept a recommendation, the user has to trust that
the recommendation has enough accuracy. There are three types of trust violations a
user consider: [15]
Exposure: Access to personal user data
Bias: Recommendation of users are being manipulated
Sabotage: The reduction of recommendation accuracy on purpose.
Key Assumptions:
The more a user trusts in the Recommendations System, the higher the user’s
usage frequency will be [18].
Analysis:
Based on the previous explanation of trust, the suspicion is that the level of trust can
either negatively or positively affect Product Sales and the User Satisfaction.
38
Consequences:
Of Product Sales:
Revenue increase
Prestige
Of User Satisfaction:
Repurchase
Loyalty
6.2.2 Impacts of Novelty/Confidence on Product Sales and User Preferences
Recommender
AlgorithmsCASES
Metrics
Sim(Items)
P-Values
TEST Relation Relation RelationRS Properties
Novelty
Confidence
Impact Business
Product Sales
Consequences
Popularity↑ Revenue
Impact User
User Preferences
Consequences
ReturnRepurchase
Figure 19: The impact of Novelty on Product Sales and User Intention
Scenario:
As mentioned in section 5.1.5, Novelty Recommendations refer to items about which
the user has not yet had knowledge. From the Business perspective, Novelty can
lead to additional Sales as the user discovers a new product which he gets excited
about and might purchase it. From the customer’s point of view, a new product might
have a positive impact on his preferences. However, if Novelty is viewed in an
isolated way, the user might only view the product but not purchase it. To be more
sure that Novelty has an effect on the user, one must consider Confidence at the
same the time.
Key Assumptions:
If a user discovers a new product and is confident with the Recommender
System (based on previous purchases) at the same time, he is more likely to
really buy the product which is positive for the User Preferences.
Analysis:
Although there was no evidence found in respected research papers, we can
conclude that if Novelty and Confidence are applied at the same time, the impact is
positive on the Product Sales.
39
Consequences:
Of Product Sales:
Increase in Revenue: The customer decides to buy the product which gives
additional revenue to the company.
Popularity: The user considers relies on the recommender system.
Of User Preferences:
Repurchase: If a user purchased a product under the condition that Novelty
and Confidence are together at the same time, then he’s likely to do the same
in future for a different product.
Return: When Novelty and Confidence are high but the user has not bought a
product yet, he might come back in the future which increases Product Views.
40
7. Analysis and Results
After the revision of the scenarios in the previous section, we arise to the point to
dedicate a chapter to analyze in details our findings and provide some observations
which are related to the correlation cases.
It is important to mention that in the previous section the scenarios presented had the
possibility to be documented by evidence found in previous research papers or case
studies. Nevertheless, there are some cases that cannot be still documented for the
moment but they are interesting to be analyzed.
In this chapter we mention briefly the cases which cannot be documented yet but that
are interesting to study in order to find more evidence and continue expanding the
correlations. Moreover, we add short observations on the cases that could be
documented in a short section.
Finally at the end of this chapter, we will summarize the relations among the different
metrics, recommender systems properties and impacts that are considered in our
seminar study.
7.1 Documented Scenarios
As we could observe in the previous correlation cases, on the one hand there is
enough evidence from the side of recommender system studies that mention a
relation between RS properties and their metrics in a technical interpretation, but on
the other hand studies from the side of consumer research field have an extensive
experience treating the impacts on users and their consequences such as user
behavior. In addition marketing in electronic business have also documented the link
between the impacts and their consequences in the business, like increment of
revenue, prestige or number or visits. The link between these different fields is
already experiencing an intersection and finding correlations becomes more visible
as the correlation cases described in our previous section.
7.2 Non-Documented Scenarios
Although there are many different recommender system (RS) properties identified
and quoted by several book references, only very few of them are mentioned or
applied in research papers. For instance, RS Properties such as accuracy,
41
confidence are among the most quoted in research papers rather than serendipity or
risk. In the recommendation section, we talk in more details about some particular
non-documented scenarios that can add a significant contribution and complement
the results of this seminar paper.
7.3 The correlation results between RS Properties Results and Impacts
The intention of this section is to show in more details our findings and potential
applications from our results and report our research contribution. Nonetheless,
before discussing our findings, applications and contributions, we would like to take
some time to come back to the figure 20, which was also shown in section 6, in order
to get back into the context and see clearly the directions of our results and
contributions.
As we described previously the three types of research papers categories found in
the course of our seminar research were mainly:
Technical Research Papers
Applied Short Research Papers
Comparison Applied Research Papers
We do not mention again the definitions of these three main identified categories but
it is important to emphasize the scope and the limit of the research paper
categories.(see figure 20)
Recommender
AlgorithmsCASES
Results
Metrics analysis
Recommender
PropertiesImpacts
TEST Relation Relation Relation
Technical Research Papers
Applied Short Research Papers
Comparison Applied Research Papers
Consequences
Relation
Figure 20: Research Paper Categories in Recommender Systems Field
Among all the categories, the research papers have in common that they could not
reach a deeper analysis with respect of the interpretation of the metrics or RS
properties with respect users or the business, they just limited to understand and
42
evaluate trade-offs on metrics to measure properties of recommenders and their
effect on the overall performance.
Moreover, based on the researcher’s profiles in the papers and their style to present
their results, we observed that they cannot make a deeper analysis on the effects of
the RS properties or metrics because they are limited on their expertise in the fields
such as Consumer Research or Marketing in Electronic Business and vice versa.
This fact is totally understandable and we are not saying that their research papers
are useless or with lack of results. Most of the researchers in the field of
recommender systems are involved in the computer sciences field and they are
somehow blind to see the correct direction to understand the effects of their
algorithms on the impact on business or users.
Based on the lack of direction, computer sciences or recommender system designers
still believe that the proper way of gaining such understanding of the impacts on
consumers or business is trying all the possible scenarios, different algorithms and
see the effects on the user or business. It is like trying to break the enigma
cryptographic machine trying all the 158 million combinations.
Maybe we exaggerated a little bit with this example but considering the scenarios
previously analyzed, we consider that understanding the proven consequences and
impacts from the side of marketing and consumer research can simplify the number
of scenarios that must be tested in order to understand better the impacts of
recommender algorithms on the users and business.
The Figure 21 suggests the direction of the recommender system design from the
side of consumer research, marketing and computer science fields.
43
Recommender
AlgorithmsCASES
Results
Metrics analysis
Recommender
PropertiesImpacts
TEST Relation Relation Corrrelation
Technical Research Papers
Applied Short Research Papers
Comparison Applied Research Papers
Consequences
Relation
Consumer Research and E-Marketing field Computer Science on Recommerder Systems field
Testing Assumptions
Suggested Direction of the Recommender System Design
Figure 21: Direction of the Recommender System Design
It is time to present our results and their interpretation in order to justify our previous
arguments. Figure 22 summarizes all the documented scenarios in one diagram:
Relations between metrics, RS properties, impacts and consequences.
RelationRS Property
Accuracy
Impact User
Online TimeKeep
Interest↑ Visits Buy
Metrics
K-Anonymity
Changing Rate
RelationRS Properties
Accuracy
Privacy
Adaptivity
Impact User
User Preference↑ Trust ↑ Visits↑ Loyalty
Metrics
MAE
RMSE
NMAE
RelationRelationRS Property
Accuracy
Impact Business
Product Views
CrossSelling
↑ Revenue Up-Selling
RelationConsequences
Consequences
Consequences
Figure 22: Relations between Metrics, RS properties and Impacts on Documented Scenarios
The Figure 22 displays all the founded documented scenarios with their relations and
their direction from the left to the right. On the one hand, fields like Marketing and
Consumer Research document extensively the relation between the impacts and the
consequences. On the other hand, computer scientists have documented extensively
the relation between the metrics and the RS properties.
44
Based on our documented scenarios the results show the missing correlation that is
present between the RS Properties and the Impacts (red arrow, figure 21). It is
important to mention that the relation found goes only in one direction (unidirectional)
and it does not go in the opposite one. For instance let’s see the case of Accuracy
causes an impact on Online Time, in one direction (left to right) we documented
already that the accuracy influences the user to remain more time on the website but
in the opposite direction (right to left) the reason that the user stays online on a
website does not necessary mean that the website has accuracy as a property.
The diagram in figure 22 helps for instance to the RS designers to know the
consequences of the metrics implemented by their recommender algorithms without
knowing the details of the consumer research theory but knowing the final
consequence effects on the user or business.
Marketers or managers for instance can select some desired consequences and
know which recommender algorithm metrics have the desired effect on the users or
business and buy or rely on a recommender algorithm that uses these metrics.
Our results show that finding the missing correlation between the RS properties and
impacts bring great benefits and increase the understanding not only of the impacts
of the business but it also reduces the complexity for recommender systems
designers to focus only on the metrics that create the desired effect on the side of
business or users.
45
8. Recommendations
The aim of the section is to offer a series of recommendations based on the previous
analysis, observations and results of this seminar document, in order that
researchers can continue in the direction as suggested in this document.
8.1 Variety in Recommender System Properties
In the meanwhile of our research paper seminar, we found several papers focusing
mostly on few RS properties such as accuracy or confidence. Testing not only the
performance of different algorithms brings a great idea of their overall performance
but also researchers should start to integrate in their research analysis where they
include more RS properties to complement their research or case studies.
In order to facilitate the test and design of new algorithms, we present research
suggestions from the marketing and consumer research side as non-documented
cases that can be tested in order to see if there is a correlation between the RS
properties and the impacts. These scenarios are based in the most documented
topics between Impacts and consequences among users and business.
8.1.1 Non-documented scenarios suggestions
The figure 23 present the relation between all the elements related to the non
documented suggested scenarios.
Metrics
SimilarityRelation
RelationRS Properties
Trust
Impact Business
Product SalesConsequences
↑Prestige↑ Revenue
Impact User
Satisfaction Consequences↑ LoyaltyRepurchase
Metrics
Sim(Items)
P-Values
Relation
RelationRS Properties
Novelty
Confidence
Impact Business
Product SalesConsequences
Popularity↑ Revenue
Impact User
User PreferencesConsequences
ReturnRepurchase
Link
Link
Figure 23: Relations between Metrics, RS properties and Impacts on non-documented Scenarios
46
The previous suggested scenarios are open to be expanded and they were
suggested from the point of view of marketing and consumer research. The
importance here is to find potential evidence that there is a correlation between the
RS properties and the impacts.
8.2 Perception of the User
Perhaps perception of the user is one of the most documented impacts in the field of
marketing and consumer research but still without a full understanding of it. As we
mentioned before, perception impact was not considered in this seminar document
due the complexity and all the wide variety of factors that involve the perception of
the user.
Future Recommender Algorithms that attempt to measure this characteristic should
be very robust and consider many different types of correlations among user
characteristics that are still not clear or still under investigation in the consumer
research field. Despite of the complexity to model this impact, the last mille in terms
of research direction in recommender systems should go in this way because once
an algorithm can predict the perception of the user, an algorithm can influence deeply
the particular human reasoning of each individual. (This situation is sometimes
documented as psychological manipulation).
8.3 New Research Questions
Meanwhile, we were analyzing and finding the different relations between the RS
properties and the impacts, we wanted to present as a part of our results a certain
kind of model that would allow us to understand in general if the success of a
recommender algorithm in terms to make a positive impact on the user or business
rely in the number of RS properties considered. Unfortunately given the nature of
research papers analyzed, they only consider a few number of RS properties and a
generalization is still too early to imply or deduce.
With the purpose to encourage future researchers, who have different knowledge
backgrounds like Consumer Research, Marketing and Computer Science, they could
look if there exist a linear or exponential behavior that can model the success of the
recommender algorithm on the impacts based on the RS properties considered.
The figure 24 shows the proposed models and the potential relation to prove
between the number of RS properties and the level of impact on users and business.
47
Level of Impact on the User
# RS Properties Considered0
Algorithm 1
Algorithm 3
Algorithm 2
Level of Impact on the Business
# RS Properties Considered0
Algorithm 1
Algorithm 3
Algorithm 2
Figure 24: Relation between RS Properties and their impact on Users and Business
48
9. Conclusion
The field of recommender systems is highly active and in constant evolution. On the
one hand researchers from the computer side field have focused strongly their efforts
to improve or create new algorithms and classify some of the their properties, but on
the other hand Consumer Research and Marketers have target their efforts to
understand the impacts and the behavior of the consumer.
At the end the final goal of the extensive study of recommender systems on the field
of electronic business is mainly focus to obtain a win-win relation between the
provider of a service (Business) and the customer.
After our exhaustive paper research, we detected an opportunity to create a good
synergy between two fields that seemed to be focusing their research in different
directions. As we presented in our results, both fields start to overlap each other and
based on this circumstances, we could find enough evidence that allowed us to
extend the interpretation of their research paper results and enhance their research
findings.
Furthermore, based on our results recommender systems designers, marketers or
consumer researchers can find easily the relations among the metrics needed in
certain scenarios to cause an assertive effect on the business or in the users as well
as they can also look at the desired consequences in order to have a clearer goal of
which algorithms should be applied.
Our seminar paper research gives a first approach of finding these correlations and
we are aware that there are much more relations to discover but it depends
enormously of the variety of RS properties tested in the algorithms. As long as the
diversity of studies in different RS properties increases, there will be more
possibilities for researchers to find more relations, which would allow them to refocus
their research in a better direction and discover interesting scenarios rather than
applying all their algorithms to all the possible scenario combinations.
49
10. Future Work
Based on the analysis, recommendations and conclusions presented in this seminar
document, it is important to mention the next steps or some research directions
derived from this paper. The next points represent a starting reference point where
researches can contribute with better interpretation results and continue with the
research of this seminar document. The next list mentions some recommendation
points derived from our research as a future work.
Researchers should start to test more variety of RS Properties in their papers,
rather than focusing on the most quoted RS properties such as accuracy and
confidence.
Continue their research of testing algorithms in the direction of testing the non-
documented scenarios proposed.
Research in the direction to find if there exists a relation between the impact of
the users or business vs the number of RS properties considered in the
algorithm
The perception of the user should start to be modeled in a more precise way
as long as the algorithms can take in consideration more properties without
compromising the performance of the algorithm.
Researchers should include Qualitative Analysis in their research.
50
11. References
[1].- Adomavicius, Gediminas, et al. "Do recommender systems manipulate consumer preferences? A study of anchoring effects." Information Systems Research 24.4 (2013): pp. 956 – 975 [2].- Chaffey, Dave. E-Business and E-Commerce Management: Strategy, Implementation and Practice. Harlow: Pearson; 2009. [3].- Ekstrand Michael D., Riedl John T., Konstan Joseph A. Collaborative Filtering Recommender Systems. Foundations and Trends in Human–Computer Interaction. Vol. 4, No. 2 (2010) 81–173 [4].- Jannach, Dietmar, et al. Recommender systems: An introduction. Cambridge University Press, 2010. [5].- Kowalczyk, Wojtek, Zoltán Szlávik, and Martijn C. Schut. "The impact of recommender systems on item, user, and ranking--diversity." Agents and Data Mining Interaction. Springer Berlin Heidelberg, 2012. 261-287. [6].- Rokach, Lior, Bracha Shapira, and Paul B.Kantor. Recommender systems handbook. Vol. 1. New York: Springer, 2011. [7].- Bobadilla et al. Recommender systems survey, in: Knowledge-Based Systems, Vol. 46 (2013), pp. 109-132 [8].- Beel, Jorean et al. Research Paper Recommender System Evaluation: A Quantitative Literature Survey, Magdeburg [9].- Lathia, N., Hailes, S., and Capra, L., The effect of correlation coefficients on communities of recommenders. In SAC ’08: Proceedings of the 2008 ACM symposium on Applied computing, pp. 2000–2005, New York, NY, USA, 2008. [10].- Joeran Beel, Stefan Langer, Marcel Genzmehr, Bela Gipp, Corinna Breitinger, and Andreas Nürnberger. Research paper recommender system evaluation: A quantitative literature survey. In Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation, RepSys '13, pages 15{22, New York, NY, USA, 2013. ACM. [11]. – Lee, Dokyun, Hosanagar, Kartik. Impact of Recommender Systems on Sales Volume and Diversity, in: Thirty Fifth International Conference on Information Systems, Auckland 2014. [12]. – Bambauer-Sache, Silke. Consumer Research 2014; Handout „Part A: Basic knowledge in the field of consumer research“ [13]. – Goldberg, Ken et al. Eigentaste: A Constant Time Collaborative Filtering Algorithm, University of California, Berkeley, IEOR and EECS Departments, 2000. [14]. - Nah, F. A study on tolerable waiting time: How long are Web users willing to wait? Behaviour & Information Technology, forthcoming, University of Nebraska-Lincoln, 2004.
51
[15]. – Karat, Marie-Claire, Blom Jan O., Karat John. Designing personalized user experiences in eCommerce, Human-Computer Iteraction Series Vol. 5, Kluwer Academic Publishers, 2004. [16]. – Lam, Shyong K “Tony”, Frankowski Dan, Riedl John. Do You Trust Your Recommendations? An Exploration Of Security and Privacy Issues in Recommender Systems, GroupLens Research, Computer Science and Engineering, University of Minnesota, 2005. [17]. – Bellogín, Alejandro et al. Report on the Workshop on Reproducibility and Replication in Recommender Systems Evaluation (RepSys), Centrum Wiskunde & Informatica, The Netherlands, Gravity R&D, Hungary, Universidad Autonoma de Madrid, Spain, 2013. [18]. - Robillard, M.P., Maalej, W., Walker, R.J., Zimmermann, Th. (Eds.) Recommendation Systems in Software Engineering. Springer Berlin-Heidelberg, 2014. [19]. - Federal Data Protection and Information Commissioner (FDPIC). Explanatory notes on Big Data, http://www.edoeb.admin.ch/datenschutz/00683/01169/index.html?lang=en, accessed on: April 29, 2015.