Distributional Off-Policy Evaluation in Reinforcement Learning
研究了多变量奖励下强化学习的分布式离策略评估问题，提出一种基于Wasserstein距离的离线方法，能同时估计多变量折扣累积回报的联合分布，并给出了有限样本误差界。
In the literature of reinforcement learning (RL), off-policy evaluation is mainly focused on estimating a value of a target policy given the pre-collected data generated by some behavior policy. Motivated by the recent success of distributional RL in many practical applications, we study the distributional off-policy evaluation problem in the batch setting when the reward is multi-variate. We propose an offline Wasserstein-based approach to simultaneously estimate the joint distribution of a multivariate discounted cumulative reward given any initial state-action pair in the setting of an infinite-horizon Markov decision process. Finite sample error bound for the proposed estimator with respect to a modified Wasserstein metric is established in terms of both the number of trajectories and the number of decision points on each trajectory in the batch data. Extensive numerical studies are conducted to demonstrate the superior performance of our proposed method.