Enhancing Exploration in Actor-Critic Algorithms: An Approach to Incentivize Plausible Novel States
提出一种基于状态新颖性和潜在收益的内在奖励方法，激励演员-评论家算法探索合理新颖状态，在五个环境中平均提升训练回报19%、降低标准差30%。
Actor-critic (AC) algorithms are model-free deep reinforcement learning techniques that have consistently demonstrated effectiveness across various domains. Enhancing exploration (action entropy) and exploitation (expected return) through more efficient sample utilization is pivotal to their success. A key strategy for a learning algorithm is to intelligently navigate the environment's state space, prioritizing the exploration of rarely visited states over frequently encountered ones. However, conventional approaches rarely quantify a novel state's utility for policy learning, which can lead to inefficient exploration. To address this, we propose an innovative approach to bolster exploration by employing an intrinsic reward based on a state's novelty and the potential benefits of exploring that state, which we term plausible novelty. Our method seamlessly integrates with off-policy AC algorithms. By incentivizing the exploration of plausibly novel states, AC algorithms can achieve substantial improvements in sample efficiency and overall training performance. Empirical results demonstrate 19% improvement in training return and 30% reduction in standard deviation, averaged across comparisons of three benchmark algorithm pairs in five different environments.