Statistically efficient advantage learning for offline reinforcement learning in infinite horizons

Shi, Chengchun ORCID: 0000-0001-7773-2099, Luo, Shikai, Le, Yuan, Zhu, Hongtu and Song, Rui (2022) Statistically efficient advantage learning for offline reinforcement learning in infinite horizons. Journal of the American Statistical Association. ISSN 0162-1459

Text (Shi_statistically-efficient-advantage-learning--accepted) - Accepted Version Download (1MB)

• Author

Abstract

We consider reinforcement learning (RL) methods in offline domains without additional online data collection, such as mobile health applications. Most of existing policy optimization algorithms in the computer science literature are developed in online settings where data are easy to collect or simulate. Their generalizations to mobile health applications with a pre-collected offline dataset remain unknown. The aim of this paper is to develop a novel advantage learning framework in order to efficiently use pre-collected data for policy optimization. The proposed method takes an optimal Q-estimator computed by any existing state-of-the-art RL algorithms as input, and outputs a new policy whose value is guaranteed to converge at a faster rate than the policy derived based on the initial Q-estimator. Extensive numerical experiments are conducted to back up our theoretical findings. A Python implementation of our proposed method is available at https://github.com/leyuanheart/SEAL

Item Type:	Article
Official URL:	https://www.tandfonline.com/journals/uasa20
Additional Information:	© 2022 American Statistical Association
Divisions:	Statistics
Subjects:	H Social Sciences > HA Statistics
Date Deposited:	19 Jul 2022 13:18
Last Modified:	20 Dec 2024 00:44
URI:	http://eprints.lse.ac.uk/id/eprint/115598

Actions (login required)

View Item