Risk-Sensitive Reinforcement Learning With Exponential Criteria
针对强化学习对噪声和参数扰动敏感的问题，提出一种基于指数准则的风险敏感强化学习方法，通过修改目标函数来近似鲁棒策略，并设计了在线Actor-Critic算法，实验表明该方法提高了样本效率和鲁棒性。
While reinforcement learning (RL) has shown experimental success in a number of applications, it is known to be sensitive to noise and perturbations in the parameters of the system, leading to high variability in the total reward amongst different episodes on slightly different environments. To introduce robustness, as well as sample efficiency, risk-sensitive RL methods are being thoroughly studied. In this work, we provide a definition of robust RL policies and formulate a risk-sensitive RL problem to approximate them, by solving an optimization problem with respect to a modified objective based on exponential criteria. In particular, we study a model-free risk-sensitive variation of the widely used Monte Carlo policy gradient algorithm, and introduce a novel risk-sensitive online Actor-Critic algorithm based on solving a multiplicative Bellman equation using stochastic approximation updates. Analytical results suggest that the use of exponential criteria generalizes commonly used ad-hoc regularization approaches, improves sample efficiency, and introduces robustness with respect to perturbations in the model parameters and the environment. The implementation, performance, and robustness properties of the proposed methods are evaluated in simulated experiments.