Visionary Policy Iteration for Continuous Control
提出远见策略迭代框架，通过构建包含后继状态的远见Q函数，提升连续动作强化学习任务的决策准确性和样本效率，并在Mujoco和OpenAI Gym任务上验证了其优于现有算法。
In this article, a novel visionary policy iteration (VPI) framework is proposed to address the continuous-action reinforcement learning (RL) tasks. In VPI, a visionary Q-function is constructed by incorporating the successor state into the standard Q-function. Due to the introduction of the successor state, the proposed visionary Q-function captures information about state transitions within the Markov decision process (MDP), thereby providing a forward-looking perspective that enables a more accurate and foresighted evaluation of potential action outcomes. The relationship between the visionary Q-function and the standard Q-function is analyzed. Subsequently, both the policy evaluation and policy improvement rules in VPI are designed based on the proposed visionary Q-function. The convergence proof for VPI is provided, ensuring that the iterative policy sequence in VPI will converge to the optimal policy. By combining the VPI framework with the twin delayed deep deterministic policy gradient (TD3) algorithm, a visionary TD3 (VTD3) algorithm is developed. The evaluation of VTD3 is performed on multiple continuous-action control tasks from Mujoco and OpenAI Gym platforms. The results of comparative experiments demonstrate that VTD3 can achieve more competitive performance than other state-of-the-art (SOTA) RL approaches. Additionally, the experimental results indicate that VPI enhances decision-making capability, reduces Q-function estimation bias, and improves sample efficiency, thereby boosting the performance of existing RL algorithms.