Optimizing pessimism in dynamic treatment regimes: a Bayesian learning approach

Zhou, Yunzhe, Qi, Zhengling, Shi, Chengchun and Li, Lexin (2023) Optimizing pessimism in dynamic treatment regimes: a Bayesian learning approach. Proceedings of Machine Learning Research, 206. ISSN 1938-7228 (In Press)

Text (Optimizing Pessimism in Dynamic Treatment Regimes: A Bayesian Learning Approach) - Accepted Version Download (1MB)

• Author

Abstract

In this article, we propose a novel pessimismbased Bayesian learning method for optimal dynamic treatment regimes in the offline setting. When the coverage condition does not hold, which is common for offline data, the existing solutions would produce sub-optimal policies. The pessimism principle addresses this issue by discouraging recommendation of actions that are less explored conditioning on the state. However, nearly all pessimism-based methods rely on a key hyper-parameter that quantifies the degree of pessimism, and the performance of the methods can be highly sensitive to the choice of this parameter. We propose to integrate the pessimism principle with Thompson sampling and Bayesian machine learning for optimizing the degree of pessimism. We derive a credible set whose boundary uniformly lower bounds the optimal Q-function, and thus we do not require additional tuning of the degree of pessimism. We develop a general Bayesian learning method that works with a range of models, from Bayesian linear basis model to Bayesian neural network model. We develop the computational algorithm based on variational inference, which is highly efficient and scalable. We establish the theoretical guarantees of the proposed method, and show empirically that it outperforms the existing state-of-theart solutions through both simulations and a real data example.

Item Type:	Article
Additional Information:	© 2023 The Author.
Divisions:	Statistics
Subjects:	H Social Sciences > HA Statistics
Date Deposited:	22 Feb 2023 10:57
Last Modified:	14 Sep 2024 09:27
URI:	http://eprints.lse.ac.uk/id/eprint/118233

Actions (login required)

View Item