Improving network dynamic pricing policies through offline reinforcement learning
针对网络动态定价中状态和动作空间指数增长导致传统方法依赖启发式的问题，提出基于离线强化学习的定价方法，仅需历史销售数据即可训练，无需在线探索或精确仿真，实验表明策略优于原有行为策略。
Abstract Due to exponentially growing state and action spaces, network dynamic pricing problems are analytically intractable such that state-of-the-art approaches rely on heuristics. Reinforcement learning has successfully been applied in various complex domains, but its successful applicability to pricing may be limited by two factors. First, the need for extensive state and action space exploration causes lost revenues when directly training within the real world. Secondly, alternatively replicating the real world in an accurate simulation to perform the training therein comes with limitations as well, because calibrating the simulation would require precise domain knowledge, which in general does not exist. To overcome the above issues, with this work, we propose a new dynamic pricing approach based on offline reinforcement learning. In contrast to online reinforcement learning, training solely requires a static data set containing information on historic sales, which stems from applying some arbitrary behavior policy in the past. In particular, we develop a low-dimensional state and actions space reformulation of the considered generic dynamic pricing problem which allows to incorporate the critic-regularized regression algorithm within a scalable approach. We also adapt the standard algorithm’s actor loss function, such that it can deal with the pricing problem’s state-dependent action space. Our studies show that the trained policy dominates and in some cases substantially outperforms the respective behavior policy. Hence, although there are some limitations that have to be discussed, offline reinforcement learning seems to be a promising approach for dynamic pricing in case online reinforcement learning is not an option.