An online sequential test for qualitative treatment effects

Shi, Chengchun ORCID: 0000-0001-7773-2099, Luo, Shikai, Zhu, Hongtu and Song, Rui (2021) An online sequential test for qualitative treatment effects. Journal of Machine Learning Research, 22. ISSN 1532-4435

Text (Shi_an-online-sequential-test-for-qualitative-treatment-effects--accepted) - Accepted Version Available under License Creative Commons Attribution. Download (1MB)

• Author

Abstract

Tech companies (e.g., Google or Facebook) often use randomized online experiments and/or A/B testing primarily based on the average treatment effects to compare their new product with an old one. However, it is also critically important to detect qualitative treatment effects such that the new one may significantly outperform the existing one only under some specific circumstances. The aim of this paper is to develop a powerful testing procedure to efficiently detect such qualitative treatment effects. We propose a scalable online updating algorithm to implement our test procedure. It has three novelties including adaptive randomization, sequential monitoring, and online updating with guaranteed type-I error control. We also thoroughly examine the theoretical properties of our testing procedure including the limiting distribution of test statistics and the justification of an efficient bootstrap method. Extensive empirical studies are conducted to examine the finite sample performance of our test procedure.

Item Type:	Article
Official URL:	https://www.jmlr.org/
Additional Information:	© 2021 The Authors
Divisions:	Statistics
Subjects:	Q Science > QA Mathematics > QA75 Electronic computers. Computer science H Social Sciences > HA Statistics
Date Deposited:	28 Oct 2021 10:06
Last Modified:	20 Dec 2024 00:42
URI:	http://eprints.lse.ac.uk/id/eprint/112521

Actions (login required)

View Item