Dual Behavior Regularized Offline Deterministic Actor–Critic
提出DBRAC算法，通过在策略评估和策略改进阶段分别施加行为正则化，缓解离线强化学习中的外推误差，在D4RL数据集上表现优于现有方法。
To mitigate the extrapolation error arising from offline reinforcement learning (RL) paradigm, existing methods typically make learned Q-functions over-conservative or enforce global policy constraints. In this article, we propose a dual behavior regularized offline deterministic Actor-Critic (DBRAC) by simultaneously performing behavior regularization on the coupling-iterative policy evaluation (PE) and policy improvement (PI) in the policy iteration process. In the PE phase, the difference between the Q-function and behavior value is first taken as the anti-exploration behavior value regularization term to drive the Q-function toward its true Q-value, which significantly reduces the conservatism of learned Q-function. In the PI phase, the estimated action variances of behavior policy in different states are then utilized for designing the weight and threshold of mild-local behavior cloning regularization term, which standardizes the local improvement potential of learned policy. Experiments on the well-known datasets for deep data-driven RL (D4RL) demonstrate that the DBRAC can quickly learn more competitive task-solving policies in various offline situations with different data qualities, significantly outperforming state-of-the-art offline RL baselines.