23 Microsoft Research CORE10 Project Summary Booklet Integrating Mechanism Design and Machine Learning Approaches for Human Computation Yuko Sakurai Kyushu University [email protected] http://yuko-sakurai.jp/projects/ms-core-10.html 1. Project Goal One of the most notable services recently introduced to the web is crowdsourcing such as Amazon Mechanical Turk (AMT). Crowdsourcing t is based on the idea of the wisdom of crowds and solves a problem by combining the efforts of many people. Using crowdsourcing services, a requester can ask many workers around the world to do her task at a relatively low cost. Crowdsourcing is also gathering attention from computer science researchers as a platform for human computation, which solves problems that can only be solved by a computer. It utilizes human intelligence as functions in computer programs. Since human computation consists of many people with different motivations and abilities, the quality control of the task results executed by human is considered as one of the most serious problems. Conventional quality control methods that introduce many redundant tasks sacrifice the economic advantage of crowdsourcing The goal of this project is to develop mechanisms for incentivizing workers who sincerely execute their tasks and truthfully declare their information including task results which are required by requesters. We integrate mechanism design and machine learning approaches. Mechanism design studies the shape of a game's rules/ protocols so that agents have an incentive to truthfully declare their preferences, and designer can select socially desirable outcomes. To design an appropriate incentive for workers, we assume that a mechanism designer (requester) has prior knowledge about the abilities of the workers in the population since most requesters repeatedly post their tasks in crowdsourcing. However, if this prior knowledge is incorrect, the mechanism will not work well. On the other hand, machine learning studies on building models from real data. In the machine learning based approach, the cold-start problem happens, since it requires a sufficient amount of data to accurately estimate models. For example, a worker has to finish a certain number of tasks before we can estimate her ability. However, new kinds of tasks and workers continuously flow in crowdsourcing services and limit the applicability of machine learning based approach. Due to the fundamental difference in the approaches, insufficient studies integrate both approaches. 2. Technical breakthrough In this project, we developed the following mechanisms to give incentives to each worker who truthfully declares her prediction/task result in crowdsourcing settings, since requesters need to avoid an untruthful worker’s behavior to obtain high quality task results (Fig. 1). Mechanisms for participatory sensing In a participatory sensing, obtaining an accurate prediction of the actions of workers is valuable for a requester who is collecting real-world information from the crowd. If an agent predicts an external event that she cannot control herself (e.g., tomorrow's weather), any proper scoring rule Figure 1 Necessity of appropriate incentives for crowdsourced workers
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
23 Microsoft Research CORE10 Project Summary Booklet
Integrating Mechanism Design and Machine Learning Approaches for Human Computation
Yuko Sakurai
Kyushu University
http://yuko-sakurai.jp/projects/ms-core-10.html
1. Project Goal
One of the most notable services recently introduced to the web is crowdsourcing such as Amazon Mechanical Turk (AMT). Crowdsourcing t is based on the idea of the wisdom of crowds and solves a problem by combining the efforts of many people. Using crowdsourcing services, a requester can ask many workers around the world to do her task at a relatively low cost. Crowdsourcing is also gathering attention from computer science researchers as a platform for human computation, which solves problems that can only be solved by a computer. It utilizes human intelligence as functions in computer programs. Since human computation consists of many people with different motivations and abilities, the quality control of the task results executed by human is considered as one of the most serious problems. Conventional quality control methods that introduce many redundant tasks sacrifice the economic advantage of crowdsourcingThe goal of this project is to develop mechanisms for incentivizing workers who sincerely execute their tasks and truthfully declare their information including task results which are required by requesters. We integrate mechanism design and machine learning approaches. Mechanism design studies the shape of a game's rules/protocols so that agents have an incentive to truthfully declare their preferences, and designer can select socially desirable outcomes. To design an appropriate incentive for workers, we assume that a mechanism designer (requester) has prior knowledge about the abilities of the workers in the population since most requesters repeatedly post their tasks in crowdsourcing. However, if this prior knowledge is incorrect, the mechanism will not work well. On the other hand, machine learning studies on building models from real data. In the machine learning based approach, the cold-start problem happens, since it requires a sufficient
amount of data to accurately estimate models. For example, a worker has to finish a certain number of tasks before we can estimate her ability. However, new kinds of tasks and workers continuously flow in crowdsourcing services and limit the applicability of machine learning based approach. Due to the fundamental difference in the approaches, insufficient studies integrate both approaches.
2. Technical breakthrough
In this project, we developed the following mechanisms to give incentives to each worker who truthfully declares her prediction/task result in crowdsourcing settings, since requesters need to avoid an untruthful worker’s behavior to obtain high quality task results (Fig. 1).
Mechanisms for participatory sensingIn a participatory sensing, obtaining an accurate prediction of the actions of workers is valuable for a requester who is collecting real-world information from the crowd. If an agent predicts an external event that she cannot control herself (e.g., tomorrow's weather), any proper scoring rule
Figure 1 Necessity of appropriate incentives for crowdsourced workers
24Microsoft Research CORE10 Project Summary Booklet
can give an accurate incentive. In our problem setting, an agent needs to predict her own action (e.g., what time tomorrow she will take a photo of a specific place) that she can control to maximize her utility. Also, her (gross) utility can vary based on an eternal event. We proved that a mechanism can satisfy our goal if and only if it utilizes a strictly proper scoring rule, assuming that an agent can find an optimal declaration that maximizes her expected utility. This declaration is self-fulfilling; if she acts to maximize her utility, the probabilistic distribution of her action matches her declaration, assuming her prediction about the external event is correct. Furthermore, we developed a heuristic algorithm that efficiently finds a semi-optimal declaration, and show that this declaration remains self-fulfilling. We also examined our heuristic algorithm's performance and describe how an agent acts when she faces an unexpected scenario.
Mechanisms for prediction tasksIn this study, we consider asks in which a worker declares predictive probability distribution over uncertain events (e.g. tomorrow’s weather in a specific location). We develop new reward plans based on strictly proper scoring rules with which a requester can flexibly determine rewards for agents to elicit truthful predictive probability distribution over a set of uncertain events. Existing strictly proper scoring rules for categorical events only reward an agent for an event that actually occurred. However, different incorrect predictions vary in quality and the requester would like to assign different reward to them, according to her subjective similarity among events, e.g. overcast is closer to sunny than to rainy. On the other side, a risk averse agent wants rewards not only for correct predictions but also for incorrect predictions. We generalize existing strictly proper scoring rules so that the requester can assign rewards for incorrect predictions based on the similarity between events. Positive definite matrices whose eigenvalues were all positive have been used for representing the similarity between examples in various machine learning methods such as kernel methods. We theoretically showed that if the reward matrix is positive definite, the generalized scoring rules are guaranteed to maximize an agent’s expected utility when she truthfully declares her prediction. We are now studying a method to empirically determine the optimal positive definite payment matrix that maximizes the accuracy of predictions from workers’ declarations.
3. Innovative Applications
Our proposed mechanisms are key technologies to realize effective and variable human computation. Especially, in crowdsourcing, a huge number of unspecified people access and work on the services and thus requesters have to determine appropriate rewards (monetary incentives) to control workers’ behaviors. We can apply our first mechanism to solve scheduling/recruiting problems in crowdsourced tasks (e.g. participatory sensing) (Fig. 2). Our second mechanism can be applied for micro tasks in crowdsourcing, prediction markets, and so on (Fig. 3).
Figure 3. Example of predicting tasks
Figure 2 Example of participatory sensing tasks
25 IJARC CORE10 project summary booklet
4. Academic Achievement
Our work has been published at a world-class conference “AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2014)” [1]. We won the JSAI Annual Conference Award at the 28th Annual Conference of the Japan Society of Artificial Intelligence (JSAI 2014) [2]. In addition, our work has been accepted at “International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2015)” [3]. We are planning to submit a paper based on the results obtained in this project at other international conferences [4,5].
5. Achievement in Talent Fostering
Two graduate students of the project investigator’s laboratory were involved. One received his master degree from the Graduate School of Information Science and Electrical Engineering, Kyushu University. The other student continues to study the research topics related to this project.
6. Collaboration with Microsoft Research
Dr. Sebastian Lahaie from MSR New York visited the principal investigator at Kyushu University and gave useful comments on the related topics of this project. We are looking forward to further collaboration with him in our future work.
7. Project Development
We are applying for several research grants such as Grant-in-Aid for Scientific Research from MEXT.
8. Publications
Paper publication1) Masaaki Oka, Taiki Todo, Yuko Sakurai, and Makoto
Yokoo, Predicting Own Action: Self-fulfilling Prophecy Induced by Proper Scoring Rules, accepted by The 2nd AAAI Conference on Human Computation and Crowdsourcing (HCOMP-2014), 2014.
2) Akira Fudo, Masaaki Oka, Taiki Todo, Yuko Sakurai, and Makoto Yokoo, Study on workers’ behaviors for effective use of limited resources, presented at the 28th Annual Conference of the Japan Society of Artificial Intelligence (JSAI-2014), 2014. JSAI Annual Conference Award.
3) Shunsuke Tsuruta, Masaaki Oka, Taiki Todo, Yuko Sakurai, and Makoto Yokoo, Fairness and False-Name Manipulations in Randomized Cake Cutting, accepted The 14th International Conference on Autonomous Agents and Multiagent Systems (AAMAS-2015), 2015.
4) Yuko Sakurai, Satoshi Oyama, Masato Shinoda, and Makoto Yokoo, Generalized Proper Scoring Rules for Flexible Reward Design to Attract Risk-Averse Agents, in preparation to submit at AAAI Conference on Human Computation and Crowdsourcing (HCOMP-2015), 2015.
5) Yuko Sakurai, Satoshi Oyama, in preparation.
Related Documents
