Recommending Knowledgeable People in a Work-Integrated Learning System (RecSysTEL Workshop at EC-TEL 2010) Günter Beham, Barbara Kump, Tobias Ley, Stefanie Lindstaedt
Jan 24, 2015
Recommending Knowledgeable People
in a Work-Integrated Learning System
(RecSysTEL Workshop at EC-TEL 2010)
Günter Beham, Barbara Kump, Tobias Ley, Stefanie Lindstaedt
October 20, 10 / 2 Executive Board Meeting, Graz
Organisa(ons try to transform workplaces into more effec(ve learning environments
[e.g., Billet, 2000]
Knowledge Workers seek for inter-personal help
I am filling out this new report form. Any idea what all these abbreviations mean?
Hmm, not really but maybe Paul could help here. He filled a similar report last week.
[Kooken et al., 2007]
Challenge: Finding knowledgeable people for a topic within a company
APOSDLE Vision
Enable learning directly at the workplace
Support people in sharing their knowledge
Reuse available resources as learning materials
The APOSDLE Approach: Connecting user activities with organisational models to recommend knowledgeable people
The APOSDLE People Recommendation Workflow
September 29, 2010 / 9
How APOSDLE looks like
October 20, 10 / 10
• Screenshots vom APOSDLE Prototypen: Suggests und Coopera;on Wizard
People
Company Resources
3-Tier Architecture of APOSDLE Services
September 29, 2010 / 11
Organisational Models
September 29, 2010 / 12
Maintaining the APOSDLE User Model September 29, 2010 / 13
Viewing a Resource Performing a Task
Being Contacted Sharing a Resource
...
0 20 40 60 80
100
Identifying Knowledge Levels
September 29, 2010 / 14
Beginner Advanced
Expert
0
20
40
60
80
100
Beginner
Advanced
Expert
Where APOSDLE Services come into play Detecting the learning need of a worker
Finding a knowledgeable person who can help
September 29, 2010 / 15
Testing and evaluating the APOSDLE User Model and Services
Simulation Study Comparison of different algorithms for maintaining the user model
Which algorithm can detect a user‘s knowledge level best?
Workplace Evaluations Deployment of APOSDLE in 2 real work environments
Comparison of knowledge level as diagnosed by APOSDLE with self-assessment
September 29, 2010 / 17
Simulation Study (Example Design)
Fixed Parameters Number of persons, number of user events, inference algorithm
Variable Parameters User behavior (Beginner, Advanced, Expert)
Level Advanced
Behavior 60% norm.
Inference Frequency
Level Beginner
Behavior 60% norm.
Inference Frequency
Level Expert
Behavior 60% norm.
Inference Frequency
September 29, 2010 / 18
Simulation Result (Example)
6 Persons 1 Topic 50 events/Behavior type Inference: Weighted Frequencies
Simulation Result (Example)
6 Persons 1 Topic 50 events/Behavior type Inference: Weighted Frequencies with windowing
Deploying APOSDLE in real workplaces
Real-world evaluation in 2 Organisations
Library of a Distance University
10 Users, only 5 Users willing to participate in the self-assessment
Used APOSDLE for 4,5 Months
Self-assessment (online questionnaire)
Innovation Management (ISN)
6 Users
Used APOSDLE for 3 Months
Self-assessment and peer-assessment (using cards)
September 29, 2010 / 22
How well does APOSDLE detect the workers‘ Work topics?
Knowledge levels?
Library of a Distance University
How well does APOSDLE detect the workers‘ work topics?
September 29, 2010 / 23
APOSDLE user model
Work Topic Non-Work Topic Total
self-assessment Work Topic 133 81 214
Non-Work Topic 4 12 16
Total 137 93 230
In many cases, APOSDLE did not „know“ that topics were a user‘s work topics
Library of a Distance University How well does APOSDLE detect the workers‘ knowledge
levels?
September 29, 2010 / 24
APOSDLE user model
Expert Advanced Beginner No Work Topic Total
Self-assessment
Expert 1 27 11 44 83
Advanced 3 39 20 31 93
Beginner 2 24 6 6 38
No Work Topic 0 4 0 12 16
Total 6 94 37 93 230
APOSDLE classified users mostly „advanced“ where they regarded themselves as „beginners“ or „experts“
Innovation Management
How well does APOSDLE detect the workers‘ work topics?
September 29, 2010 / 25
APOSDLE user model
Work Topic (%) Non-Work Topic (%) Total
self-assessment Work Topic 356 (41.7) 334 (39.0) 690 (80.7) Non-Work Topic 51 (6.0) 114 (13.3) 165 (19.3) Total 407 (47.7) 448 (52.3) 855 (100)
In many cases, APOSDLE did not „know“ that topics were a user‘s work topics
September 29, 2010 / 26
Number of user interactions with APOSDLE
The more interaction with APOSDLE, the more correct detections of work topics and non-work topics
Innovation Management How well does APOSDLE detect the workers‘ knowledge
levels?
APOSDLE user model
Expert Advanced Beginner No Work Topic Total
Self-assessment
Expert 27 130 29 162 348 Advanced 11 73 19 82 185 Beginner 7 49 11 90 157 No Work Topic 5 36 10 114 165
Total 50 288 69 448 855
APOSDLE classified users mostly „advanced“ where they regarded themselves as „beginners“ or „experts“.
September 29, 2010 / 27
Discussion of Outcomes
In many cases, APOSDLE was not able to identify a user‘s work topics Users NEVER dealt with this topic within APOSDLE
Evaluation period too short? Rather: not enough system usage during evaluation period
In many cases, APOSDLE erroneously diagnosed „advanced“ level Improve algorithms
Self-assessment may also be erroneous/biased Better „external measure“ for workplace evaluations??
September 29, 2010 / 28
Outlook
Improving algorithms for diagnosing user knowledge Cross-validation with existing data
Further evaluations of the user model in other organisations
Combination of different recommendation strategies
Evaluating People Recommendations Evaluation Setup?
Lab studies
Field studies
September 29, 2010 / 29