A robust test for the stationarity assumption in sequential decision making

Wang, Jitao, Shi, Chengchun ORCID: 0000-0001-7773-2099 and Wu, Zhenke (2023) A robust test for the stationarity assumption in sequential decision making. Proceedings of Machine Learning Research. pp. 36355-36379. ISSN 1938-7228

Text (A Robust Test for the Stationarity Assumption in Sequential Decision Making) - Accepted Version Download (690kB)

• http://chrome-extension://efaidnbmnnnibp...

Abstract

Reinforcement learning (RL) is a powerful technique that allows an autonomous agent to learn an optimal policy to maximize the expected return. The optimality of various RL algorithms relies on the stationarity assumption, which requires time-invariant state transition and reward functions. However, deviations from stationarity over extended periods often occur in real-world applications like robotics control, health care and digital marketing, resulting in suboptimal policies learned under stationary assumptions. In this paper, we propose a model-based doubly robust procedure for testing the stationarity assumption and detecting change points in offline RL settings with certain degree of homogeneity. Our proposed testing procedure is robust to model misspecifications and can effectively control type-I error while achieving high statistical power, especially in high-dimensional settings. Extensive comparative simulations and a real-world interventional mobile health example illustrate the advantages of our method in detecting change points and optimizing long-term rewards in high-dimensional, non-stationary environments.

Item Type:	Article
Official URL:	https://proceedings.mlr.press/v202/
Additional Information:	© 2023 The Author
Divisions:	Statistics
Subjects:	Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Date Deposited:	17 Nov 2023 10:03
Last Modified:	20 Dec 2024 00:48
URI:	http://eprints.lse.ac.uk/id/eprint/120775

Actions (login required)

View Item