Cookies?
Library Header Image
LSE Research Online LSE Library Services

Off-policy confidence interval estimation with confounded Markov decision process

Shi, Chengchun, Zhu, Jin, Shen, Ye, Luo, Shikai, Zhu, Hongtu and Song, Rui (2022) Off-policy confidence interval estimation with confounded Markov decision process. Journal of the American Statistical Association. ISSN 0162-1459

[img] Text (COPE-JASA) - Accepted Version
Repository staff only until 12 August 2023.

Download (915kB) | Request a copy

Abstract

This paper is concerned with constructing a confidence interval for a target policy’s value offline based on a pre-collected observational data in infinite horizon settings. Most of the existing works assume no unmeasured variables exist that confound the observed actions. This assumption, however, is likely to be violated in real applications such as healthcare and technological industries. In this paper, we show that with some auxiliary variables that mediate the effect of actions on the system dynamics, the target policy’s value is identifiable in a confounded Markov decision process. Based on this result, we develop an efficient off-policy value estimator that is robust to potential model misspecification and provide rigorous uncertainty quantification. Our method is justified by theoretical results, simulated and real datasets obtained from ridesharing companies. A Python implementation of the proposed procedure is available at https://github.com/Mamba413/cope.

Item Type: Article
Additional Information: © 2022 The Author(s).
Divisions: Statistics
Date Deposited: 03 Aug 2022 08:42
Last Modified: 05 Sep 2022 14:12
URI: http://eprints.lse.ac.uk/id/eprint/115774

Actions (login required)

View Item View Item

Downloads

Downloads per month over past year

View more statistics