Off-policy evaluation in doubly inhomogeneous environments

Bian, Zeyu, Shi, Chengchun ORCID: 0000-0001-7773-2099, Qi, Zhengling and Wang, Lan (2024) Off-policy evaluation in doubly inhomogeneous environments. Journal of the American Statistical Association. ISSN 0162-1459

Text (Off-Policy Evaluation in Doubly Inhomogeneous Environments) - Published Version Available under License Creative Commons Attribution. Download (1MB)

• Author
• Scopus publication

Abstract

Abstract–: This work aims to study off-policy evaluation (OPE) under scenarios where two key reinforcement learning (RL) assumptions—temporal stationarity and individual homogeneity are both violated. To handle the “double inhomogeneities”, we propose a class of latent factor models for the reward and transition functions, under which we develop a general OPE framework that consists of both model-based and model-free approaches. To our knowledge, this is the first article that develops statistically sound OPE methods in offline RL with double inhomogeneities. It contributes to a deeper understanding of OPE in environments, where standard RL assumptions are not met, and provides several practical approaches in these settings. We establish the theoretical properties of the proposed value estimators and empirically show that our approach outperforms state-of-the-art methods. Finally, we illustrate our method on a dataset from the Medical Information Mart for Intensive Care. An R implementation of the proposed procedure is available athttps://github.com/ZeyuBian/2FEOPE. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

Item Type:	Article
Additional Information:	© 2024 The Author(s)
Divisions:	Statistics
Subjects:	H Social Sciences > HA Statistics
Date Deposited:	21 Aug 2024 09:24
Last Modified:	20 Dec 2024 00:56
URI:	http://eprints.lse.ac.uk/id/eprint/124630

Actions (login required)

View Item