This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Practical guidelines for designing and evaluating educationally oriented recommendations
Olga C. Santos, Jesus G. Boticario
Abstract
There is a need for designing educationally oriented recommendations that deal with educational
goals as well as learners’ preferences and context in a personalised way. They have to be both based
on educators’ experience and perceived as adequate by learners. This paper compiles practical
guidelines to produce personalised recommendations that are meant to foster active learning in
online courses. These guidelines integrate three different methodologies: i) user centred design as
defined by ISO 9241-210, ii) the e-learning life cycle of personalised educational systems, and iii)
the layered evaluation of adaptation features. To illustrate guidelines actual utility, generality and
flexibility, the paper describes their applicability to design educational recommendations in two
different e-learning settings, which in total involved 125 educators and 595 learners. These
applications show benefits for learners and educators. Following this approach, we are targeting to
cope with one of the main challenges in current massive open online courses, which are expected to
provide personalised education to an increasing number of students without the continuous
involvement of educators in supporting learners during their course interactions.
them are more appropriate for their own learning. Here, it can be taken advantage of the well-
grounded research on recommender systems. In particular, recommender systems can be used to
guide users in a personalised way to useful objects in a large space of possible options (Burke,
2002) reducing the existing information overload. This is framed in the so-called personalisation
task of adaptive navigation support in educational scenarios (Brusilovsky and Peylo, 2003).
When recommendations are designed in educational scenarios, they should involve learners in
the learning process, and thus, suggest carrying out actions that foster their learning performance
(i.e., ensuring the accomplishment of given educational goals). It is noticeable that the most
common approach followed by educational recommender systems mainly focuses on pointing
learners to “read relevant resources” –as a mere information retrieval issue (Drachsler et al., 2015) –
and not on taking advantage of available recommendation opportunities that require actual
involvement of learners through other “potential actions” that can be done in the course space, as
suggested in early approaches (Zaïane, 2002).
The design of recommendations in general has not received much attention in related literature
so far. In fact, it has been neglected in the field of recommender systems. By and large, the focus
has been put on evaluating the performance of the recommendation algorithms (i.e. the analysis of
an algorithm's runtime in practice) in terms of information retrieval measures, such as accuracy,
recall, precision and so on (Konstan and Riedl, 2012). In this sense, there have been some efforts to
identify descriptions of domain-independent tasks in recommender systems, with the goal to help
distinguish among different evaluation measures (Herlocker et al., 2004).
The closest effort that we are aware of related to recommendations design in educational
scenarios is the repertory grid from the personal construct theory proposed by Kelly (1955), which
has been used by (Hsu et al., 2010) to develop reading material recommendations from domain
knowledge elicited from multiple experts. Here, recommendations are provided to the system by the
educators. Additionally, Brito et al. (2012) have proposed an architecture-centred solution for
designing educational recommender systems in a systematic manner. However, we have not find in
the literature approaches that address in educational scenarios the design and evaluation of
educationally oriented recommendations.
Within the educational arena, the spectrum of recommendation opportunities cannot be
considered just as an information retrieval issue. Here eliciting and using educators’ background on
attending learning needs may be crucial when catering for the learner’s needs in a given situation.
However, educators are not provided with guidelines that help them to designing and evaluating
educationally oriented recommendations that result from their experience in attending learners and
which may support adaptive navigation paths within online courses. Actually, as it will be discussed
later on, there is a gap found between literature demands (recommendations should focus on the
learning needs and foster active learning) and literature outcomes (educational recommender
systems actually deliver mainly learning contents).
To deal with this issue, there are three different methodologies that can be considered: i) user
centred design as defined by ISO 9241-210, ii) the e-learning life cycle of personalised educational
systems, and iii) the layered evaluation of adaptation features.
Bearing all this in mind, in order to support the process of developing educationally oriented
recommendations, this paper presents a set of design and evaluation practical guidelines for three
specific iterations of the recommendation design and evaluation cycle (i.e., proof of concept,
elicitation of recommendations and delivery of recommendations). Resulting recommendations are
to be delivered to learners through a semantic educational recommender system -SERS (Santos and
Boticario, 2011a), which is in line with the service-oriented approach of the third generation of
learning management systems (Dagger et al., 2007) where external educational web based services
can interoperate with the learning management systems (Muñoz-Merino et al., 2009). SERS rely on
i) a recommendation model, ii) an open standard-based service-oriented architecture, and iii) a
usable and accessible graphical user interface to deliver the recommendations. The proposed
guidelines implement the recommendation cycle and focus on identifying recommendation
opportunities that come out from studying the teaching and learning issues that characterise the
3
educational domain. Thus, they require the involvement of educators and learners in that
identification process. To evaluate the benefits of applying these guidelines, we report their usage in
two very different educational contexts, which consider different approaches and have different
requirements. These contexts involve two different learning scenarios implemented in two different
learning management systems. In particular, this paper describes how those guidelines have been
applied in these two scenarios, involving a total of 125 educators and 595 learners.
Accordingly, this paper is structured as follows. First, we comment on related works that can
guide the design and evaluation of educational recommendations. Here the focus is on the users’
involvement in the design process and formative evaluation of adaptation features. Afterwards, we
present the set of practical guidelines, which are meant to support educators in designing and
evaluating user centred recommendations for educational scenarios. Next, we report on the
application of these guidelines in two very different contexts: 1) the DtP course in dotLRN learning
management system, and 2) the EBIFE course in Willow free-text adaptive computer assisted
assessment system. After discussing the approach and results provided, we conclude summarising
the main issues involved, and introduce current work, which extends the features considered in
these contexts to support educators in eliciting recommendations that account for affective issues.
2. Related works
The utility of recommender systems for the educational domain has been largely acknowledged
over the last fifteen years as a way to provide personalised support to learners while carrying out
learning tasks in web-based learning environments (Drachsler et al., 2015). Research has shown that
recommendations to be provided in the educational domain are different from those in other
domains (e.g., e-commerce, e-entertainment). In fact, there are a number of distinctive issues when
educational recommendations are compared with recommendations for consumers, mainly in terms
of goals, user features and recommendation conditions (Draschler et al., 2009a). Therefore,
recommender systems should not be transferred from commercial to educational contexts on a one-
to-one basis, but rather need adaptations in order to facilitate learning (Buder and Schwind, 2012).
In that respect, there are long-running challenges derived from the peculiarities of the educational
domain (Konstan and Riedl, 2012).
When recommendations are designed for educational scenarios a distinctive factor is, for
instance, that they should not be guided just by the learners’ preferences (Tang and McCalla, 2009).
Considering only users’ preferences as the bases for providing recommendations is typically done
in non-educational recommenders (Kluver et al., 2012). However, a personalisation support in
educational settings has to deal with diverse learning styles and other psycho-educational aspects of
the learning process (Bates and Leary, 2001; Blochl et al., 2003), as well as the cognitive state of
the learner (Drachsler et al., 2009a). In this context, the benefit of providing recommendations to
learners is to be related to improvements on their performance in the course, through a more
effective, efficient and satisfactory learning (Drachsler et al., 2009b). In other words, all these
conditions affect the design (knowledge modelling), development (techniques, algorithms and
architectures) and evaluation (in real world e-learning scenarios) of recommender systems in
education (Santos and Boticario, 2012).
In order to identify key issues to be considered when designing educational recommendations,
an extensive review of related literature covering 50 recommender systems has been carried out
elsewhere (Santos and Boticario, 2013). This review shows that despite the first approaches
remarked on recommending several types of actions (e.g., accessing a course notes module, posting
a message on the forum, doing a test, trying a simulation) and resources (Zaïane, 2002), most
research have neglected this possibility and systems have mainly focused on recommending some
specific item of content. In fact, it shows that there are very few examples of educational
recommender systems that foster users’ active actions (e.g., providing a contribution). However,
current educational approaches acknowledge the benefits of learners’ active involvement in her
learning process (Lord et al., 2012) and it is noticeable that fostering interaction can promote
collaboration with like-minded learners (Wang, 2007) and improve learning (Webb et al., 2004).
4
As to the involvement of users, user centred design methodologies can be followed to develop
systems that suit users’ needs (Gulliksen et al., 2003). Thus, to cater for the learner in terms of
suitable educational recommendations according to their needs, it is suggested to incorporate user
centred design methodologies in the recommendations development process covering both their
design and evaluation. However, none of the 50 systems reviewed (Santos and Boticario, 2013)
reported the application of a methodology that involves users in the design process to find out
relevant recommendations opportunities for their educational scenarios. Furthermore, regarding the
evaluation, an extended review with a total of 59 systems showed that there were only 7 works
which evaluated the effect on the learning performance of the recommendations delivered (Santos
et al., 2014a).
It has also been suggested that for context-rich domains (like the educational one), end-users
and stakeholders should be provided with tools for expressing recommendations that are of interest
(Adomavicius et al., 2011). In this way, educators need to be provided with some mechanism that
allows them to experiment designing recommendations to be delivered to their learners. This might
help them to cope with the wide variety of potential recommendation opportunities that exist in the
learning environments (Bieliková et al., 2014) and which have not yet been sufficiently explored.
Thus, interactions between actors (learners, educators, etc.), artefacts and environment make up a
process from where to understand the learning issues involved, evaluate the educational result and
support the design of effective technology (Gassner et al., 2003).
In this sense, involving domain experts in the recommendations generation process can produce
more accurate recommendations (Shen and Shen, 2004; Al-Hamad et al., 2008) as these can
reproduce educators’ decision-making behaviours (Hsu et al., 2010). Educators with wide
experience in on-line teaching have a comprehensive view of the difficulties encountered by
learners. Thus, they can put these difficulties in perspective as regards to the seriousness and
frequency of the issue for the learners (Hanover Research Council, 2009). Moreover, learners can
also be involved in the recommendation process to design and evaluate educational resources (Ruiz-
Iniesta et al., 2012) or to adapt parameters and recommendation algorithms (Farzan and Brusilovsky,
2006). In fact, the learner involvement in the recommendation process can have benefits related to
satisfaction and trust (Buder and Schwind, 2012). Therefore, to cope with aforementioned issues it
is sensible to consider that both learners (i.e., users) and educators (i.e., designers) have to be
involved in the recommendations development process.
From the above findings follows both that users should be involved from the beginning in an
educationally oriented recommendation elicitation process and that to this, they have to be
supported. However, despite knowing that learning is a personalised and evolving process that is to
be focused on the learner and regardless the benefits of applying user centred design to the
development of adaptive learning systems (Gena, 2006), user centred design is usually neglected.
Unfortunately, when developing adaptive learning systems, users are generally consulted (if at all)
towards the end of the development cycle (Harrigan, et al., 2009), forgetting that the design process
should be focused on the learner and not on the system (Mao et al., 2005). In fact, specific user
centred design methodologies are needed when the user's goals involve learning and teaching
(Gamboa Rodriguez et al., 2001).
In this respect, ISO 9241-210 ‘Ergonomics of human-system interaction - Part 210: Human-
centred design for interactive systems’ (ISO, 2010) is the international standard that sets the basis
for many user centred design methodologies. It is generic and can be applied to any interactive
system or product. This standard describes four principles of human-centred design: 1) active
involvement of users (or those who speak for them), 2) appropriate allocation of function (making
sure human skill is used properly), 3) iteration of design solutions (therefore allowing time in
project planning), and 4) multi-disciplinary design (but beware overly large design teams). Here,
user centred design is described as an iterative cycle, having as input the design plan that compiles
the underlying needs and requirements. Although ISO 9241-210 does not specify any methods, a
wide variety of usability methods that can be used to support user centred design are outlined in the
technical report ISO/TR 16982:2002 ‘Ergonomics of human-system interaction—Usability methods
5
supporting human-centred design’ (ISO, 2002). As the offer is wide, several initiatives have
researched into the most appropriate methods for user centred design such as the UsabiliyNet
European project (Bevan, 2003), the IDEO method cards (IDEO, 2003) and the Usability Body of
Knowledge (UXPA, 2012). Anyway, existing initiatives provide hints to select the most appropriate
methods to apply, but their selection has to be done taking into account the particularities of the
design environment, the context of use and the stage of the design process. Moreover, according to
the UsabilityNet project there are some conditions that should be taken into account in this decision:
i) limited time and/or resources to apply the methods, ii) availability of direct access to users, and
iii) limited skills and expertise of the people in charge of applying the methods.
As to how user centred design can be used to guide the production of educationally oriented
recommendations, first it has to be pointed out that user centred design relies on an iterative
development cycle that involves the user throughout the process. It leads to the definition of a set of
user requirements, and then guides the development of systems with built-in capabilities to provide
a good user experience. Actually, some usability methods have already been used to develop
recommender systems in non-educational domains (Zins et al., 2004). In the educational domain,
the e-learning life cycle has been proposed to support learner-centred adaptive educational scenarios.
It consists of four consecutive phases: design, publication, usage and auditing (Van Rosmalen et al.,
2004). In particular, to place the learner as the centre of this e-learning cycle, accessibility and
usability issues are to be taken into account throughout all the cycle phases (Martin et al., 2007).
So as to guide the development process, this user centred design iterative cycle calls for
formative evaluations (which address issues during the development or improvement of a system)
aimed at ensuring that results truly meet the user requirements identified during the design (Gena
and Weibelzahl, 2007). Moreover, recommender systems are interactive systems that offer an
adaptive output (i.e., personalised recommendation). As recommendations are to adapt its response
to the users’ needs, this adaptive support has also to be considered in the formative evaluation
(Mulwa et al., 2011). Literature has identified difficulties in evaluating adaptive systems (Van
Velsen et al., 2008). To overcome these difficulties, the adaptation process can be decomposed into
its constituents -called layers-, and each of these layers evaluated separately where necessary and
feasible (Paramythis et al., 2001). This approach can be used to evaluate the advantages of the
adaptation provided (Karagiannidis and Sampson, 2000) and guide the development process
(Paramythis et al., 2001). In this way, more can be learnt about what causes success or failure in the
adaptive response. The purpose here is to figure out why, and under what conditions, a particular
type of adaptation can be applied to achieve a specific goal.
The most up to date layered evaluation framework is the one proposed by Paramythis et al.
(2010). This work is a revised version of a previous combination carried out on three previous
frameworks (Weibelzahl and Lauer, 2001; Paramythis et al., 2001; Brusilovsky et al., 2004). This
revision defines five layers, corresponding to the main stages of adaptation. The layers are domain
independent and their relevance and application depends on the nature of the system. The layers
identified in the framework are the following: 1) Collection of input data: assembles the user
interaction data along with any other data available related to the interaction context; 2)
Interpretation of the collected data: provides meaning for the system to the raw input data
previously collected; 3) Modelling of the current state of the world: derives new knowledge about
the user, the interaction context, etc. and introduces that knowledge in the dynamic models of the
system; 4) Deciding upon adaptation: given a particular state of the world, as expressed in the
models maintained by the system, identifies the necessity of an adaptation and selects the
appropriate one; and 5) Applying (or instantiating) adaptation: introduces the adaptation in the user-
system interaction, on the basis of the related decisions. Moreover, after this piece-wise evaluation,
the framework also considers the evaluation of the adaptation as a whole, which is meant to get the
big picture. In this case, the application domain has to be taken into account to formulate and select
the appropriate evaluation metrics and methods. Up to now, the layered evaluation method has not
been applied to recommender systems, but it has been suggested as a powerful technique in
identifying areas of recommender systems that require more focused future work (Pu et al., 2012).
6
Nevertheless, there exist several open issues towards the standardization of the layered evaluation
frameworks applied to recommender systems (Manouselis et al., 2014).
Furthermore, besides the evaluation of the adaptation mechanism, in recommender systems it is
also necessary to conduct empirical evaluations that consider the entire process of how the user
experience comes about in the recommendation cycle (Knijnenburg et al., 2012). This means that
sumative evaluations (which are conducted after the system’s development and its purpose is to
provide information on the system’s ability to do what it was designed to do) are to be carried out.
From the review reported in this section follows three key issues, namely, (1) there is a need to
consider educational issues in recommender systems in education, (2) users should be involved in
designing educationally oriented recommendations, and (3) there is a lack of having practical
guidelines that help users in such design and corresponding formative evaluation. In this paper we
provide educators with guidelines to help them in designing and evaluating personalised
recommendations for their learners, which consider their learning needs, preferences and
educational context. To this, we argue that there is educators’ tacit knowledge obtained over years
of experience in supporting learners during their learning within online learning environments that
can be obtained following those guidelines. In particular, the proposed guidelines should integrate
the aforementioned methodologies that have come out in this study of related work, namely: 1) the
user centred design defined by ISO-9241-210, involving both educators and learners in the process,
2) the e-learning life cycle of personalised educational systems, and 3) the layered evaluation
approach to guide the formative evaluation of the adaptation features design. The guidelines should
also support empirical evaluations of the user experience along the recommendation process.
3. Practical guidelines
The practical guidelines that we propose have been identified from previous experience in several
research projects on technology enhanced learning and inclusion, namely aLFanet: IST-2001-33288
(Boticario and Santos, 2007), ADAPTAPlan: TIN2005-08945-C06-01 (Boticario and Santos, 2008),
CISVI: TSI-020301-2008-21 (Santos et al., 2010) and EU4ALL: IST-2006-034778 (Boticario et al.,
2012). They combine three methodological approaches: 1) user centred design following the
standard ISO 9241-210, 2) the four phases of the e-learning life cycle for developing personalised
educational systems, and 3) the layered evaluation approach that is required to formatively evaluate
the design of adaptive features for these systems.
The user centred design methodology that can support educators in identifying educationally
oriented recommendation opportunities in online courses has been defined elsewhere (Santos and
Boticario, 2011b) and is called TORMES. TORMES stands for Tutor-Oriented Recommendations
Modelling for Educational Systems. Its goal is to support educators in identifying recommendation
opportunities in learning environments that have an educational purpose, and which are perceived
as adequate by learners, both in content and time of delivery. It combines user centred design
methods and data mining analysis. Data mining techniques are used to extract information from
learners’ interactions and, from this, discover usage patterns. TORMES drives the recommendations
design in three consecutive steps: 1) elicitation of educationally sound recommendations validated
by users (i.e. educators and learners) with a collaborative review, 2) acquisition and validation of
the learners’ features to select the appropriate recommendations for the current context, and 3)
analysis of the recommendations provided and evaluation of their impact on the user. TORMES
follows the four user centred design activities defined by ISO 9241-210 in an iterative manner: 1)
Understanding and specifying the context of use: identifying the people who will use the system,
what they will use it for, and under what conditions they will use it; 2) Specifying the user
requirements: identifying any requirements or user goals that must be met for the system to be
successful, considering the variety of different viewpoints and individuality; 3) Producing design
solutions to meet user requirements, which can be made in stages to encourage creativity, from an
initial rough concept to a final full-blown design; and 4) Carrying out user-based evaluation of the
design against the requirements.
7
As discussed in Section 2, since recommendations are to enrich the adaptive support in
technology enhanced learning scenarios, they have to be managed along the e-learning life cycle of
adaptive educational systems. In previous research (i.e., the aLFanet project), it was concluded that
in order to provide a personalised learning experience, it is desirable that the actors involved in the
learning process (i.e. learners and educators) are supported during the e-learning life cycle, which
covers the following phases (Van Rosmalen et al. 2004): Design: deals with the preparation in
advance of the learning experience; Publication: manages the administration of the environment
where the learning experience is to be carried out; Use: focuses on the usage of the e-learning
environment services by learners and educators; and Auditing: provides feedback to the course
author on the learners’ experiences. The application of the e-learning life cycle in other projects
such as ADAPTAPlan and EU4ALL, showed that the recommendation process can flow along the
four phases of the e-learning life cycle (Santos, 2009) as follows: 1) the design phase covers the
generation of semantic educationally oriented recommendations described in terms of the
recommendation model; 2) the publication phase involves loading the recommendations generated
in the previous phase so that they can be instantiated through the e-learning services available in the
given e-learning environment; 3) the use phase delivers recommendations whose semantic
description matches the current runtime context, and monitors the interactions of the learners within
the e-learning environment; and 4) the auditing phase provides feedback on the recommendations
design by analysing the results on their usage over the course experience.
Since the stages of the e-learning life cycle (i.e. design, publication, usage and auditing) are to
be considered in the design of the recommendations, they should be integrated within the activities
of the user centred design interaction cycle as defined by ISO 9241-210 and thus, considered
explicitly in the user centred design cycle of the TORMES methodology. To cope with this, we
propose to split the design and formative evaluation activities of the ISO standard into two sub-
activities. In this way, the user centred design activity ‘Producing design solutions to meet user
requirements’ is broken down into two sub-activities, which correspond to the design and
publication phases of the e-learning life cycle. The rationale for this is to explicitly consider the
mapping of the recommendations needs elicited into the recommendation model proposed, and its
publication in the e-learning environment to be ready for the next activity (i.e. the formative
evaluation). These two sub-activities are defined as follows: 1) Modelling: application of the
recommendation model to semantically characterise the recommendations in terms of a semantic
recommendation model, and 2) Publication: instantiation of recommendations described with the
model into the learning environment that is going to be used to deliver the recommendations. This
recommendation model allows bridging the gap between recommendations' description provided by
the educator and the recommender logic, which is in charge of delivering recommendations in the
running course. These recommendations can be defined along the dimensions of “6 Ws and an H”
(Santos et al., 2014b): i) What (i.e., the type) is to be recommended, that is, the action to be done on
the object of the e-learning service (for instance, to post a message in the forum); ii) How and
Where (i.e,. the content) to inform the learner about the recommendation, which in a multimodal
enriched environment, should describe the modality and way in which the recommendation has to
be delivered to the learner; iii) When and to Who (i.e., the runtime information) the recommendation
is produced, which depends on defining the learner features, interaction agent capabilities and
course context that trigger the recommendation. It describes both the restrictions that may limit
recommendation delivery as well as the applicability conditions that trigger the recommendations;
iv) Why (i.e., the justification) a recommendation has been produced, providing the rationale behind
the action suggested; and v) Which (i.e., the recommendation features) additional semantic
information characterise the recommendations themselves (e.g., relevance, category). In turn, the
ISO activity ‘Evaluating designs against requirements’ is also broken down into another two sub-
activities, which correspond to the use and auditing phases of the e-learning life cycle. In this case,
the rationale behind is to explicitly separate the participation of the learners to produce the
interactions from the analysis of these interactions. These two sub-activities are defined as follows:
1) Usage: provide support to the learners when interacting within the course space by delivering
8
personalised recommendations, and 2) Feedback: carry out analysis of these interactions to
evaluate the recommendations and provide feedback to the design.
Another issue to be considered is the formative evaluation of the recommendations. As
discussed in Section 2, layered evaluation approaches are appropriate for this. Thus, to cope with
the evaluation of the design of the system adaptation features, the five typical layers of the layered
evaluation approach can also be mapped into the user centred design cycle of TORMES. In this way,
the layered evaluation approach is integrated with the activities identified in ISO 9241-210 as
follows: first (layer 1), the evaluation of the data collected is produced during the activity
‘Feedback’, as it is there where the analysis of the learners’ interactions with the recommendations
takes place. Second (layer 2), the evaluation of the interpretation of the data collected is done in
the activity ‘Understanding and specify the context of use’ of the following iteration, as the
interpreted data are used in this activity to get more insight into the context that complements the
information gathered from the learners with typical usability methods. Third (layer 3), the
evaluation of the modelling of recommendations with respect to the current state of the world in terms of the recommendation model, which is based on the data collected and interpreted, is done
in the activity Modelling of the following iteration. Fourth (layer 4), the evaluation of the strategy
selected to deliver the recommendations design upon adaptation is done in the activity
‘Publication’, when the recommendations are implemented and arranged in order to be ultimately
delivered to the learners. Fifth (layer 5) and last, the evaluation of the application of the
adaptation decisions (i.e. the delivery of the recommendations) is done in the activity ‘Usage’,
where the recommendations were instantiated and delivered to the learners in the environment.
The combination of the above three methodological approaches (i.e., user centred design as
defined in TORMES, e-learning life cycle and layered evaluation) results in the practical guidelines
compiled in Table 1. In order to drive the users along the development process, they cover three
typical iterations of the recommendation design and evaluation cycle: 1) proof of concept to
evaluate recommendations’ perception by the users, 2) elicitation of educational recommendations
derived from the practical experience of educators, and 3) delivery of the recommendations in a
large scale study. These interactions address different goals, require different input and produce
different output. They also specify the methods to consider in each of the activities, as well as the
expected outcomes for each of them. Although they suggest methods to use, the whole range of
usability methods1
are still applicable if needed, so as to keep the required flexibility and
adaptability to meet given particularities.
Iteration 1: Proof of
concept
Iteration 2: Elicitation of
recommendations
Iteration 3: Delivery of
recommendations
Iteration
Goal
Guide educators in
understanding the needs for
recommendations in e-
learning scenarios and
demonstrate the value of
extending the adaptive
navigation support in learning
environments with
recommendations.
Produce diverse educationally
oriented recommendations for a
given e-learning scenario and
focus on the perception of the
recommendations previous to
their delivery to final users in the
learning environment.
Offer learners the
recommendations elicited to find
out how the user experience
comes about in the
recommendation process and
understand their behaviour in
order to decide if
recommendations need to be
redesigned (i.e. formatively
evaluate them). Thus, it
constitutes an empirically study
of the recommendations
behaviour.
Iteration
Input
Design plan. Either the output from iteration
‘Proof of concept’ (if available)
or the design plan.
A set of recommendations
modelled and validated, usually
as a result of the iteration
‘Elicitation of educational
1 An exhaustive list of available user centred design methods can be consulted in the UsabilityNet website, the outcome
of the same name European project that provides user centred design resources to practitioners
9
Iteration 1: Proof of
concept
Iteration 2: Elicitation of
recommendations
Iteration 3: Delivery of
recommendations recommendations’.
Context
of use
Methods: meetings with
stakeholders
Outcomes: context specified
Evaluation layer: n/a
Methods: individual interviews,
questionnaires
Outcomes: redefined/adjusted
context of use and info to
produce scenarios
Evaluation layer: interpretation
of data collected in proof of
concept or externally (layer 2)
Methods: individual interviews
Outcomes: revised context of use
Evaluation layer: interpretation
of data in previous feedback
activity iteration (layer 2)
UC
D a
ctiv
itie
s +
e-l
earn
ing
lif
e c
ycl
e p
ha
ses
com
bin
ed
User
requi-
rements
Methods: brainstorming, user
observational studies, Wizard
of Oz
Outcomes: adaptation
requirements
Evaluation layer: n/a
Methods: scenario based
approach
Outcomes: scenarios of use with
educational sound
recommendations proposed in
them
Evaluation layer: n/a
Methods: focus group, interview
Outcomes: revised scenarios and
recommendations
Evaluation layer: n/a
Model-
ling of the
design
solution
Methods: modelling process
Outcomes: sample
recommendations semantically
modelled
Evaluation layer: modelling
the current state of the world
regarding recommendations
elicited and described in terms
of the model (layer 3)
Methods: focus group and card
sorting, modelling process
Outcomes: revised list of
modelled recommendations and
adjustments to the semantic
recommendation model
Evaluation layer: modelling the
current state of the world
regarding recommendations
elicited and described in terms of
the model (layer 3)
Methods: modelling process
Outcomes: revised modelling for
recommendations
Evaluation layer: modelling the
current state of the world
regarding recommendations
elicited and described in terms of
the model (layer 3)
Publi-
cation of
the design
solution
Methods: instantiation of
recommendations, pilot study
Outcomes: sample
recommendations
contextualised in the
environment
Evaluation layer: deciding
upon adaptations by checking
the applicability conditions
(layer 4)
Methods: instantiation of
recommendations, pilot study
Outcomes: technically
validation of the
recommendations
Evaluation layer: deciding
upon adaptations by checking
the applicability conditions
(layer 4)
Methods: instantiation of
recommendations, pilot study
Outcomes: educational
recommendations contextualised
in a large scale setting
Evaluation layer: deciding upon
adaptations by checking the
applicability conditions (layer 4)
Usage to
gather
evalua-
tion data
Methods: paper prototype,
storyboard, Wizard of Oz, user
observational studies
Outcomes: interaction data
from users on sample
recommendations
Evaluation layer: applying
adaptation decisions observing
system logs (layer 5)
Methods: functional prototype,
Wizard of Oz, card sorting
Outcomes: recommendations
rating and classification by users
Evaluation layer: applying
adaptation decisions by
analysing the value of
recommendations delivery
considered by the users (layer 5)
Methods: functional prototype,
observation studies
Outcomes: interaction data from
learners on educational
recommendations
Evaluation layer: applying
adaptation decisions observing
system logs (layer 5)
Feedback
from
evalua-
ting
design
requirem
ents
Methods: questionnaires,
interviews, data log analysis
Outcomes: users feedback on
the sample recommendations
Evaluation layer: collection
of data from users’ interaction
with recommendations and
feedback (layer 1)
Methods: descriptive statistics
Outcomes: feedback on
recommendations to identify the
most relevant for the given
context
Evaluation layer: collection of
data from users’ interaction with
recommendations and feedback
(layer 1)
Methods: data log, interviews,
questionnaires, significant testing
Outcomes: recommendations
feedback on empirical validation
showing recommendations that
need to be redesigned
Evaluation layer: collection of
data (layer 1)
10
Iteration 1: Proof of
concept
Iteration 2: Elicitation of
recommendations
Iteration 3: Delivery of
recommendations
Iteration
Output
A set of sample
recommendations that reflect
identified educational needs,
which made up a first mock-
up that can be shown to users
(educators/learners) and
tested. They translate
researcher ideas.
A set of validated
recommendations ready to be
delivered in the learning
management system and thus to
be formatively evaluated in a
large scale evaluation.
The identification of those
recommendations that need to be
redesigned because they did not
meet the educational objectives
proposed.
Table 1 Practical guidelines for the three iterations defined of the recommendation design and evaluation cycle
So as to clarify the issues involved in Table 1, next there is a more detailed description of the
activities considered in each of the three iterations. At the end of this section, these iterations are
summarised in Figure 1. In Section 5, we comment on the educational contexts where these
practical guidelines have been applied to cope with these iterations. More details on those contexts
are provided in some related works that report the educational scenarios where TORMES has been
applied for these three iterations: ‘Proof of Concept’ in DtP-dotLRN context (Santos and Boticario,
2010), ‘Elicitation of Educational Recommendations’ in DtP-dotLRN context (Santos and Boticario,
2013) and in EBIFE-Willow context (Pascual-Nieto et al., 2011; Santos et al., 2014a), and
‘Delivery of Recommendations’ in EBIFE-Willow context (Santos et al., 2014a). However, these
works do not include the combined methodological approach which results from integrating user
centred design, e-learning life cycle and layered evaluation within the practical guidelines that are
presented in this paper. The added value of this paper lies on both identifying and compiling
guidelines for designing learner centred educationally oriented recommendations and describing
how these guidelines can be applied to follow the methodologies that cover the recommendation
design and evaluation cycle.
The following subsections (3.1, 3.2 and 3.3) focus on describing the issues involved in Table 1
from a conceptual viewpoint, following the iterations and activities depicted afterwards in Figure 1
(Section 3.4). The examples provided in Section 4 are expected to clarify those conceptual issues
that might not be understood without an application context.
3.1 Iteration ‘Proof of Concept’
The objective of the iteration for the Proof of Concept is to come up with a preliminary research
idea as soon as possible. Thus, simple (and readily applicable) user centred design methods are the
most appropriate. Regarding the activity Context of use (Ctx1), when a proof of concept is carried
out, it is assumed that there are no reference systems available wherein potential recommendation
needs were previously identified (see related work in Section 2), and thereby there is no common
ground that can be used to get feedback from the users. As a consequence, the starting point to
define the context of use should be to review related approaches (mainly research ideas) from the
literature and previous experiences (e.g. researchers’ own experience as well as that of relevant
stakeholders). Assumptions resulting from the context of use should be validated. To this,
researchers can share their thoughts and get feedback from people who will use the system on two
key issues i) for what it will be used, and ii) under what conditions. For this, a meeting with
stakeholders can be of value, as it is a strategic way to collect information about the purpose of the
system and the overall context of use.
The activity User requirements (Req1) should focus on identifying the adaptation requirements
for the system within the context of use specified in the previous activity. The methods to be
applied for the requirement specification should allow users to come up with creative ideas on what
adaptation features are required. There are some methods that can provide valuable information,
such as a) brainstorming, which can be used to generate ideas for a given problem in a creative way,
b) user observational studies, which can be carried out with learners to see how they currently
interact in the environment where the recommender system is planned, and c) the Wizard of Oz
11
(Dahlbäck et al., 1993), which can be of practical use to clarify the logic behind as it enables
unimplemented technology to be evaluated by using a human to simulate the response of a system.
In the activity Modelling (Mod1) several recommendations can be proposed based on the
outcomes from the previous activity (e.g., the analysis of the results obtained from the user’s
interactions and the outcomes of the brainstorming) and modelled in terms of the semantic
recommendation model. Accordingly, the corresponding evaluation layer that has to be addressed is
the evaluation of the modelling of the state of the world regarding the recommendation needs
identified (third layer).
In the activity Publication (Pub1) a set of sample recommendations can be contextualised and
instantiated in the environment so that these recommendations can be tested with learners in the
following activity. When appropriate (i.e., adaptation capabilities are already provided in the
learning environment), the decision mechanism of the adaptation should be evaluated (fourth layer).
This can be done, for instance, with a pilot study to test the delivery of the recommendations after
they have been instantiated in the environment.
The activity Usage (Us1) deals with learners’ interactions with recommendations. If there is no
adaptation logic available, methods like paper prototypes, storyboards or the Wizard of Oz are quite
useful to present samples of recommendations to the learners and allow them to interact with the
recommendations and get feedback. In this case, as the adaptation effects may not be available,
when presenting the recommendations to the learner, those effects should be highlighted to her, so
that she can picture them and give her opinion. In turn, if the prototype is functional, user
observational studies can be carried out. In the latter, where the recommendations are offered in a
running prototype, the application of the adaptation decisions should be evaluated (fifth layer).
The activity Feedback (Fdb1) analyses the interactions done by learners. Explicit feedback
from the users can be gathered through questionnaires or interviews. Moreover, data log analysis
(e.g., through data mining) can be used to get knowledge from the interactions. If data from
interactions are collected and analysed, the first layer (i.e. collection of input data) approach should
be applied here to evaluate the feedback gathered. These collected data can be compared with the
results from the observational study carried out in the activity User requirements to analyse the
impact of adding the given recommendations. Furthermore, past experiences previous to the user
centred design approach can be also analysed and compared here within the design being tested.
An application of the user centred design approach as defined by TORMES for the iteration
“Proof of Concept” (Iteration 1 in Table 1) in the DtP-dotLRN context is reported elsewhere
(Santos and Boticario, 2010). The main outcomes of the application of the practical guidelines
proposed in this paper (which combine user centred design, e-learning life cycle and layer
evaluation methodological approaches) are summarised in Section 4.
3.2 Iteration ‘’Elicitation of educational recommendations’
As more factual and objective information about the effects of design decisions is expected in this
iteration, the methods suggested here require more resources (time and participants) than in the
previous iteration.
The current knowledge about the context of use (either from the previous iteration or from the
design plan) can be enriched in the activity Context of use (Ctx2). To this, individual interviews
with educators interested in using the recommendations in their courses as well as questionnaires
for a larger sample of educators can be carried out. The interview script should include relevant
questions that can be used in the next activity (see below) to obtain meaningful educational
scenarios from the educator responses. These interviews should focus on obtaining information that
reflects the type of problems encountered by learners, their goals and also positive experiences.
Specific emphasis should be made to get particularities of the learners’ profile or the course context
in order to gather information to support the system adaptation features. Data mining outcomes
from previous experiences of the educators with their learners can be provided in the interview to
support the educators’ argumentation (e.g., identifying association rules for actions carried out by
successful learners; see Section 4). If these data collected from previous experiences are discussed
12
here, then their interpretation is to be validated as defined by the second layer from the layered
evaluation approach (i.e., interpretation of the collected data).
The goal of the activity User requirements (Req2) is to give instruments to extract knowledge
from the educators on what the requirements are for the recommendations within the context of use.
Scenario-based methods (Rosson and Carroll, 2001) can be used. These scenarios consist in
involving the user in writing stories (i.e., scenarios) about the problems taking place in relevant
situations that come to their mind. Scenarios can be produced with the information obtained from
the interviews of the previous activity. Two types of scenarios are to be produced. The first one,
called problem scenario, should specify how educators carry out their tasks in the given context and
the problems identified in them, but they should not address what system features are to be used.
They are expected to cover a wide range of situations and diverse adaptation contexts and include
problematic issues that will test the system concept. The solution scenario, in turn, has to replace
those issues identified in the problem scenario with potential recommendations that can avoid them
and which are characterised in terms of the semantic recommendation model. To illustrate how
these scenarios are defined, some examples are provided in Section 4 (in particular, in Table 3).
In the activity Modelling (Mod2), a focus group can be used to involve several educators in
validating the recommendations elicited from the scenarios obtained in the previous activity and
refining the modelling done to them. In order to prepare the participants for the focus group aimed
at discussing the recommendations produced during the previous task and make them aware of the
recommendations list produced, they can be asked to individually rate the recommendations
obtained and categorise them with a card sorting method (Spencer, 2009). The purpose here is to
verify if the structure in which the educators expect the recommendations to be classified fits with
the classification proposed in the semantic recommendation model. To carry out the card sorting
activity, each of the recommendations defined should be written on a small card. Participants are
then requested to sort these cards into clusters according to their own educational criteria for
classification. If possible, an open card sorting (i.e. without a predefined set of categories) is
preferred, as this will help to depict the educators’ mental model without any bias. Categories can
be obtained with a hierarchical cluster analysis (Dong et al., 2001). After the focus group discussion,
a revised list of educational sound recommendations properly modelled is to be obtained. As a result
of the above tasks, the recommendation model may be readjusted (e.g., new attributes might be
identified to characterise the recommendations). Here, the evaluation of the modelling of the state
of the world regarding the recommendation needs identified (third layer) takes place.
In the activity Publication (Pub2), recommendations are to be instantiated to support their
eventual delivery to the learners. As in this iteration the focus is on the elicitation process, it has to
be checked if the information used to model the recommendations can be obtained from the
learning environment. Once the information involved in the modelling of the recommendations is
available, the recommendations have to be instantiated. When appropriate (i.e., the system has
adaptive capabilities), the decision mechanism of the adaptation should be evaluated (fourth layer),
for instance with a pilot study to test the delivery of the recommendations after they have been
instantiated in the learning environment.
The activity Usage (Us2) deals with the interactions with the recommendations designed.
Educators and learners are requested to interact with these recommendations in order to rate their
utility and classify them (with a closed card sorting) to find out what is their perception about them.
Participants can be given access to a running system where some sample recommendations are
offered, so they can get an idea of the meaning of a running recommendation. The running
prototype can be a functional system or a Wizard of Oz. In the former instance, the application of the
adaptation decisions should be evaluated (fifth layer).
In the activity Feedback (Fdb2), the results from the previous sub-activity are to be collected
and analysed with descriptive statistics. If interactions are gathered from the system and mined, the
first layer of the layered evaluation approach has to be applied (i.e. collection of input data). As a
result, a validated set of educationally oriented recommendations to be applied in the scenarios
elicited is obtained. These recommendations have been mapped into the model, and have been
13
validated from the users’ point of view. If the evaluation results are satisfactory, they are ready to
be delivered in the e-learning environment.
The results of applying user centred design as defined by TORMES for the iteration “Elicitation
of educational recommendations” (Iteration 2 in Table 1) are reported elsewhere for the two
different contexts, namely DtP-dotLRN (Santos and Boticario, 2013) and EBIFE-Willow (Pascual-
Nieto et al., 2011, Santos et al., 2014a). In this paper we rather focus on summarising in Section 4
the main outcomes of applying the practical guidelines compiled in Table 1 (which combine user
centred design, e-learning life cycle and layer evaluation methodological approaches).
3.3 Iteration ‘Delivery of Recommendations’
When evaluation results from previous iteration are not satisfactory, a formative empirical study
involving a large scale evaluation of the system as a whole can be carried out to obtain indicators
about the recommendations design and thus, understand their effect on the learner. This is meant to
get enough data for a meaningful statistical analysis. This study can also be seen as a rehearsal to
the summative evaluation that should be done when the whole system is finished. The methods
suggested here require more resources (time and participants) than in the previous iteration so that
statistical tests of significance can be applied.
The activity Context of Use (Ctx3) revises the current knowledge already obtained in the
previous iteration through interviews with educators complemented with the analysis of the data
collected from the interactions in the system during the previous iteration. In that case, the second
layer of the layered evaluation approach is applied as the data collected is interpreted.
To improve the activity Requirements specification (Req3), the scenarios and the
recommendations from the previous iteration can be revised with the updated context information in
a focus group or through individual interviews with educators.
In the activity Modelling (Mod3), the modelling of the recommendations revised in the previous
activity has to be checked in order to identify if there were suggested changes that are not covered
by the recommendation model. Thus, the evaluation of the modelling of the state of the world
regarding the recommendation needs identified (third layer) is carried out.
In the activity Publication (Pub3), recommendations modelled have to be instantiated in the
system so that they are ready to be used by the learners in a large scale setting. If appropriate,
especially if some of the applicability conditions depend on data mined or follow a rule-based
approach, the decision mechanism of the adaptation should be evaluated (fourth layer), for instance,
in a pilot study.
In the activity Usage (Us3), learners can interact in a functional prototype with the
recommendations obtained. Here, the above recommendations are offered when the conditions
defined in the recommendation model occur. User observational studies as well as experiments on
functional prototypes can be carried out. To evaluate the adaptation decisions applied, the fifth layer
has to be considered.
In the activity Feedback (Fdb3), the learners’ outcomes are to be collected and analysed from
data logs, questionnaires and interviews. If possible, the impact of recommendations should be
compared with an execution of the course without recommendations. The purpose here is to analyse
if the application of the recommendations has made a statistically significant impact or not. The
goal of the analysis is to find out those recommendations that did not perform well in the formative
evaluation, and thus, they need to be redesigned. Significant testing can be of help in this analysis.
As the learners’ interactions are to be collected, the first layer to evaluate the collection of input
data applies here.
An application in the EBIFE-Willow context of user centred design as defined by TORMES for
the iteration “Delivery of recommendations” (Iteration 3 in Table 1) is reported elsewhere (Santos
et al., 2014a). In this paper, we rather focus on summarising in the next Section the main outcomes
of the application of the practical guidelines proposed in this paper (which combine user centred
design, e-learning life cycle and layer evaluation methodological approaches).
14
3.4 Summary
The layout of the iterations and activities compiled in Table 1 and described in sections 3.1, 3.2 and
3.3 is shown in Figure 1.This figure explicitly introduces the phases of the e-learning life cycle and
the evaluation layers into the set of activities of the ISO-9241-210 user centred design cycle.
Previous interactions, which did not follow the user centred design approach, can be considered in
the activity Feedback of the first iteration. Furthermore, at the end of each iteration, the outcome is
to be checked to see if the system satisfies the specified design requirements. Once the iterations in
the user centred design are finished (e.g., after the third iteration in Figure 1), the recommendations
are ready to be evaluated empirically in a summative study. The red hexagons are used to point out
where the different layers of the layered evaluation approach apply.
Figure 1. Extended user centred design (UCD) cycle to support design and formative evaluation of
recommendations along the e-learning life cycle. Abbreviations used: Ctx: Context of use; Exp: Previous
experiences not following the UCD approach; Fdb: Feedback; Ly: Evaluation layer; Mod: Modelling; Pub:
Publication; Req: User requirements; Sys; System requirements satisfaction; Us: Usage.
4. Evaluating practical guidelines applicability
The practical guidelines proposed for the three iterations of the recommendation design and
evaluation cycle were applied in two very different contexts so as to evaluate their suitability to deal
with diverse situations in real-world online educational scenarios. These scenarios involved
contexts that differ both on the learning (different learning setting and contents) and the
technological side (different learning platform), as commented below.
The first context (DtP-dotLRN) corresponds to the course ‘Discovering the Platform’ (DtP).
This course has been developed following the ALPE methodology (Santos et al., 2007b), which
15
produces accessible Sharable Content Objet Reference Model (SCORM) 1.2 compliant courses and
is designed following the approach of learning by doing (Schank and Cleary, 1995), which means
that simple activities are defined to make use of the different platform services. It teaches how to
use the dotLRN platform to novice users. dotLRN is an open source collaboration oriented learning
management system, which was originally developed at the Massachusetts Institute of Technology
(MIT) and used in universities worldwide for its accessibility support, technological flexibility and
interoperability capabilities. For these reasons it has been the main learning platform considered
over the last decade in the research of the aDeNu group (Santos et al., 2007a).
The goal in the second context (EBIFE-Willow) is to offer a full e-learning course through a
learning system initially designed for blended learning (i.e. combining face to face teaching and
computer-based education). This system is Willow, a free-text computer assisted assessment system
that allows students to answer open-ended questions in natural language (Pérez-Marín et al., 2009).
In this context, two educators with experience in using Willow in blended learning settings had the
need for teaching the MOOC on ‘Search strategies in the Web with Educational Goals’ (EBIFE as
abbreviated in Spanish). The objective for integrating recommendations in Willow was to widen
Willow’s usage to a full e-learning context, where the physical presence of the educator was not
available (Pascual-Nieto et al., 2011). Here, recommendations are required to provide adaptive
navigation support in order to guide the learners in their interaction, covering those navigational
issues that were solved by the educators in the face-to-face session to introduce Willow. This is
meant to foster a proactive attitude of the learner which facilitates the usage of Willow without the
educator support. Thus, the design process has to embed the educators’ way of supporting their
learners during the course interaction as the idea is that the recommendations play the role of the
educator when she is not available. Details on how to provide adaptive navigation support in
Willow with recommendations involving an interdisciplinary team of software developers and