Testing stationarity and change point detection in reinforcement learning

Li, Mengbing, Shi, Chengchun ORCID: 0000-0001-7773-2099, Wu, Zhenke and Fryzlewicz, Piotr ORCID: 0000-0002-9676-902X (2025) Testing stationarity and change point detection in reinforcement learning. Annals of Statistics. ISSN 0090-5364 (In Press)

Text (Testing stationarity and change point detection in reinforcement learning) - Accepted Version Pending embargo until 1 January 2100. Available under License Creative Commons Attribution. Download (2MB)

Abstract

We consider reinforcement learning (RL) in possibly nonstationary environments. Many existing RL algorithms in the literature rely on the stationarity assumption that requires the state transition and reward functions to be constant over time. However, this assumption is restrictive in practice and is likely to be violated in a number of applications, including traffic signal control, robotics and mobile health. In this paper, we develop a modelfree test to assess the stationarity of the optimal Q-function based on pre-collected historical data, without additional online data collection. Based on the proposed test, we further develop a change point detection method that can be naturally coupled with existing state-of-the-art RL methods designed in stationary environments for online policy optimization in nonstationary environments. The usefulness of our method is illustrated by theoretical results, simulation studies, and a real data example from the 2018 Intern Health Study. A Python implementation of the proposed procedure is publicly available at https://github.com/limengbinggz/CUSUM-RL.

Item Type:	Article
Additional Information:	© 2025 The Author
Divisions:	Statistics
Subjects:	H Social Sciences > HA Statistics
Date Deposited:	05 Mar 2025 14:30
Last Modified:	05 Mar 2025 14:33
URI:	http://eprints.lse.ac.uk/id/eprint/127507

Actions (login required)

View Item