Adaptive Rewards Mechanism for Sustainable Online Learning Community Julita Vassileva Computer Science Department University of Saskatchewan, Canada
Feb 09, 2017
Adaptive Rewards Mechanism for Sustainable Online Learning Community
Julita Vassileva
Computer Science Department University of Saskatchewan, Canada
Outline
• The problem • Comtella: a link-sharing online community• Encouraging students to rate links• Adaptive rewards mechanism
– Community model and individual model– Adaptive rewards
• Case study• Conclusions
The problem• Why encouraging participation in online communities?
– The majority of systems remain unused– Incentive mechanism is needed
• To encourage participation • However, excessive participation can kill a community
• To ensure sustainability the incentive mechanism needs to:– Discourage excessive participation– Ensure a way to measure the quality of contributions– Encourage timely contributions (when needed by the
community most) and good contributions (from people who tend to have higher standards)
Comtella
• Brief history– 2002: P2P (Gnutella) client for sharing research
papers (as files) in the MADMUC lab (Vassileva @ CoopIS’02, Bretzke & Vassileva @ UM’03)
– 2004: Modified to support a 4th year students in a class on Ethics and IT to share URLs related to each topic/ week) – (Cheng & Vassileva @ ITS2004, @HICSS05)
– 2005: Modified as centralized web-based online community (try it out at http://svaroy.usask.ca:8080/comtella)
back
Motivation Strategies in Comtella• Rewarding participatory acts with points and status
(membership)– Sharing new links, rating links– Points accumulate and result in higher membership for the user
– Visualization of the community showing user participation allowing different views for each kind of activity
• Results - case study in 2004: – Very effective in increasing participation in sharing new papers– But also some low quality papers; excessive participation, high
cognitive loadCheng R., Vassileva, J. (2005) User Motivation and Persuasion Strategy for Peer-to-peer Communities. HICSS'2005 Vassileva, J. (2004) Harnessing P2P Power in the Classroom. Proceedings Intelligent Tutoring Systems, ITS'2004
Summary so far
• Measuring and rewarding participation can stimulate participation
• However, quality tends to deteriorate (some students game the system)
• Therefore, – excessive participation should not be
encouraged – a way for measuring quality and rewarding
high-quality contributions is needed
Encouraging students to rate links
• Modeled after Slashdot– Students with high membership get more
ratings to give out – more power• In addition, each act of rating is rewarded
– By earning c-points (a kind of virtual currency)– By earning participation points (towards
higher membership)
Measuring the Quality of Contributions
• All contributed URLs start at 0 – unlike Slashdot, where it depends on the
“karma” of the user who submitted the post
• The final rating is the sum of all received ratings
• How to judge the quality of a rating?– based on its similarity with other ratings for the
same link
Incentives for rating (1)
• Each act of rating earns c-points for the user• The user can “invest” her c-points to increase
the visibility of the links she contributes
Incentives for rating (2)
• By rating links users also earn participation points towards their membership
• The number of points earned depends on the quality of each rating– Tendency towards the “average” taste?– Is this the taste of the community?
• How to prevent gaming the system?– allow sorting search results by average rating – but do not show the average rating
Adaptive Rewards Mechanism
• Adapting to what?– To the current needs of the community – To the individual quantity and quality
standards of the user• Adapting what?
– The participation points earned by different activities (“economic” incentives)
– The personal message to the user at login
Skip details
Community Model
• Expected / desirable number of links for topic (set by instructor) – Qc– Depends on topic, the time
in the semester, busyness of students etc.
• Community reward factor (depends on time since introducing topic) - Fc
Individual Model (1)• The user reputation in bringing high quality
links: RI = the average summative rating of all links contributed by the user – Users with high RI should be encouraged to
contribute more at any time– Users with low RI should be discouraged, unless
the topic is still fresh and lacks links. • The expected number of links from individual
user for the current topic
I
ICI R
RQQ
Individual model (2)
• Individual reward factor FI
• User reputation in giving high-quality ratings– Computed as 1/EI where
N
RRE
N
iii
I
1
Adaptive rewards mechanism• Varying weights Wi(t) for particular forms of
participation • The user contribution is computed as:
where t – the time of contribution
• The weights depend on the state of the user’s individual model and on the community model at the moment of contribution– Weight for sharing links – Wieght for giving ratings – proportional to the user
reputation for giving high quality ratings 1/EI
n
i
T
ti
i
tWV1 1
)(
ICSS FFWW 0
Adaptive messages to the user
• Communicating the rewards that the user will get for each type of action at the time of login.– user who shares many papers of poor quality
gets a low RI and a small QI , therefore little reward for subsequent contributions.
– the related message would be to contribute less in next period but improve the quality of her contributions.
– Show it
Case study• Comtella used in the “Ethics and IT” class at the UofS
– Jan-April 2005– 32 students
• Two groups of 16 throughout the term• Test - Comtella 1: with adaptive rewards mechanism and c-points• Control - Comtella 2: no adaptive rewards mechanism, no c-points
• Groups formed to have equal gender and Canadian / foreign representation– Test and Control groups are separate communities
• no interaction of shared links, ratings etc. • however, students were in the same classroom for lectures; project teams
across both groups• We compared the numbers of contributions in each group (links,
ratings)• Post-study online questionnaire
Questions and answers (1)• Did the users in the test group (Comtella 1) give more
ratings? – Yes: nearly twice as much as Comtella 2: 1065 vs. 613 ratings (significant)
• Did the summative ratings in Comtella 1 reflect better the quality of the contributed links?– Yes: in Comtella 1, 56% (9 users) felt that the final summative ratings that
their links received reflect fairly their quality, while in Comtella 2, only 25% (4 users) thought so.
• Did the users in Comtella 1 tend to share links earlier in the week?– Yes: users in Comtella 1 shared 71.3% of their contributions in the first 3
days after introducing the topic; users in Comtella 2 shared 60.6% of their contributions in the first 3 days. The difference was significant for all topics and ranged between 7-14%.
• Did the users in Comtella 1 share the number of links that was expected from them? – To some extent: about half of the users did.
Questions and answers (2)• Did the users in Comtella 1 participate more actively in general?
– Yes: they read more papers (3419 vs. 2416) and logged in the system more frequently (1714 vs. 982).
• Is there a significant difference with respect to the total number of contributed links between the test and the control group? – No: 613 in Comtella 1 versus 587 in Comtella 2– There was no excessive paper contribution in either case.
• Did the users in Comtella 1 contribute links with higher quality?• Is there a correlation between the ratings of links and the times they
were chosen for summarizing?– Yes: strong correlation: – 0.861 for links with rating > 0,
0.928 for links with rating >1, 0.983 for links with rating > 2, 1 for links with rating > 3
Discussion• Incorporating an incentive mechanism can stimulate a
desired behaviour in an online community – in our case, the c-points stimulated ratings– can be useful for other systems depending on user ratings, e.g.
collaborative filtering systems (Movie Lens, Amazon)• An adaptive rewards mechanism can orchestrate a desired
pattern of collective behaviour– in our case, the time-adaptation of the rewards stimulated users to
make contributions earlier• It is important to make the user aware of the rewards for
different actions at any given time• More experiments are needed to show effect of adaptation
to individual pattern and impact on the quality / quality perception
Future work
• Deal with the subjectivity of ratings• Adapt the mechanism to motivate
contributions in a different kind of community, – a research group or a group of graduate
students sharing research papers, – a community of researchers (the AIED and /or
the UM community?) – a community for girls and women in science
and engineering
More information
• Try Comtella at: – http://svaroy.usask.ca:8080/comtella– See how it is used for sharing research papers in the
MADMUC lab: http://svaroy.usask.ca:8080/aied
• See other papers, follow up project:– http://bistrica.usask.ca/madmuc/peer-motivation.htm– Or just Google for “Comtella”
Back