Optimal stopping and worker selection in crowdsourcing: an adaptive sequential probability ratio test framework

Li, Xiaoou, Chen, Yunxiao ORCID: 0000-0002-7215-2324, Chen, Xi, Liu, Jingchen and Ying, Zhiliang (2021) Optimal stopping and worker selection in crowdsourcing: an adaptive sequential probability ratio test framework. Statistica Sinica, 31 (1). 519 - 546. ISSN 1017-0405

Text (Optimal stopping and worker selection in crowdsourcing) - Accepted Version Download (337kB)

• Author
• Related Item

Abstract

In this study, we solve a class of multiple testing problems under a Bayesian sequential decision framework. Our work is motivated by binary labeling tasks in crowdsourcing, where a requestor needs to simultaneously choose a worker to provide a label and decide when to stop collecting labels, under a certain budget constraint. We begin by using a binary hypothesis testing problem to determine the true label of a single object, and provide an optimal solution by casting it under an adaptive sequential probability ratio test framework. Then, we characterize the structure of the optimal solution, that is, the optimal adaptive sequential design, which minimizes the Bayes risk using a log-likelihood ratio statistic. We also develop a dynamic programming algorithm to efficiently compute the optimal solution. For the multiple testing problem, we propose an empirical Bayes approach for estimating the class priors, and show that the average loss of our method converges to the minimal Bayes risk under the true model. Experiments on both simulated and real data show the robustness of our method, as well as its superiority over existing methods in terms of its labeling accuracy.

Item Type:	Article
Official URL:	http://www3.stat.sinica.edu.tw/statistica/
Additional Information:	© 2021 Institute of Statistical Science, Academia Sinica
Divisions:	Statistics
Subjects:	H Social Sciences > HD Industries. Land use. Labor H Social Sciences > HD Industries. Land use. Labor > HD28 Management. Industrial Management H Social Sciences > HA Statistics
Date Deposited:	23 May 2019 15:18
Last Modified:	04 Oct 2024 03:42
URI:	http://eprints.lse.ac.uk/id/eprint/100873

Actions (login required)

View Item