Cookies?
Library Header Image
LSE Research Online LSE Library Services

Optimal stopping and worker selection in crowdsourcing: an adaptive sequential probability ratio test framework

Li, Xiaoou, Chen, Yunxiao, Chen, Xi, Liu, Jingchen and Ying, Zhiliang (2020) Optimal stopping and worker selection in crowdsourcing: an adaptive sequential probability ratio test framework. Statistica Sinica. ISSN 1017-0405 (In Press)

[img] Text (Optimal stopping and worker selection in crowdsourcing) - Accepted Version
Pending embargo until 1 January 2100.

Download (337kB) | Request a copy

Identification Number: 10.5705/ss.202018.0300

Abstract

In this paper, we aim at solving a class of multiple testing problems under the Bayesian sequential decision framework. Our motivating application comes from binary labeling tasks in crowdsourcing, where a requestor needs to simultaneously decide which worker to choose to provide a label and when to stop collecting labels under a certain budget constraint. We start with a binary hypothesis testing problem to determine the true label of a single object, and provide an optimal solution by casting it under the adaptive sequential probability ratio test (Ada-SPRT) framework. We characterize the structure of the optimal solution, i.e., the optimal adaptive sequential design, which minimizes the Bayes risk by making use of a log-likelihood ratio statistic. We also develop a dynamic programming algorithm that can efficiently compute the optimal solution. For the multiple testing problem, we further propose to adopt an empirical Bayes approach for estimating class priors and show that our method has an averaged loss which converges to the minimal Bayes risk under the true model. The experiments on both simulated and real data show the robustness of our method and its superiority in labeling accuracy comparing with several other recently proposed approaches

Item Type: Article
Official URL: http://www3.stat.sinica.edu.tw/statistica/
Additional Information: © 2019 Institute of Statistical Science, Academia Sinica
Divisions: Statistics
Subjects: H Social Sciences > HD Industries. Land use. Labor
H Social Sciences > HD Industries. Land use. Labor > HD28 Management. Industrial Management
H Social Sciences > HA Statistics
Date Deposited: 23 May 2019 15:18
Last Modified: 29 May 2020 23:07
URI: http://eprints.lse.ac.uk/id/eprint/100873

Actions (login required)

View Item View Item

Downloads

Downloads per month over past year

View more statistics