News REcommendation Evaluation Lab (NewsREEL) Results Frank Hopfgartner, Benjamin Kille, Andreas Lommatzsch, Martha Larson, Torben Brodt, Jonas Seiler
News REcommendation Evaluation Lab (NewsREEL)
Results
Frank Hopfgartner, Benjamin Kille, Andreas Lommatzsch, Martha Larson, Torben Brodt, Jonas Seiler
Registrations
2
Task 1: Online Evaluation
• Provide recommendations for visitors of the news portals of plista’s customers
• Ten portals (local news, sports, business, technology)
• Communication via Open Recommendation Platform (ORP)
Dat
a
• Benchmark own performance with other participants and baseline algorithms during three pre-defined evaluation windows
• Best algorithms determined in final evaluation period
• Standard evaluation metricsEva
luat
ion
Recommend news articles in real-time
Task 2: Offline Evaluation
• Traffic and content updates of nine German-language news content provider websites
• Traffic: Reading article, clicking on recommendations
• Updates: adding and updating news articlesD
ata
• Simulation of data stream using Idomaar framework
• Participants have to predict interactions with data stream
• Quality measured by the ratio of successful predictions by the total number of predictionsE
valu
atio
n
Predict interactions in a simulated data stream
Evaluation Schedule (Task 1)
First evaluation window
Second evaluation window
Third evaluation window
CTR (Task 1)
9
Response rate vs number of requests (Task 1)
10
Switched off throughout the day
Error rate (Task 1)
11
Click-through rate
12
Congratulations to „xyz-2.0“ for achieving highest
CTR with 1.17%
Availability/response rate
13
Congratulations to „flumingsparkteam“ for
achieving highest response rate with
99.64%
Presentations