Risk-Sensitive Deep RL: Variance-Constrained Actor-Critic Provably Finds Globally Optimal Policy
研究了平均奖励设定下带方差约束的风险敏感深度强化学习，提出一种演员-评论家算法，证明其能以次线性速率找到全局最优策略，并通过仿真验证。
While deep reinforcement learning has achieved tremendous successes in various applications, most existing works focus on maximizing the expected value of total return and ignore its inherent stochasticity. Such stochasticity is also known as the aleatoric uncertainty and is closely related to the notion of risk. This work makes the first attempt to study risk-sensitive deep reinforcement learning under the average reward setting with the variance risk criteria. Particularly, we focus on a variance-constrained policy optimization problem where the goal is to find a policy that maximizes the expected value of the long-run average reward, subject to a constraint that the long-run variance of the average reward is upper bounded by a threshold. Utilizing Lagrangian and Fenchel dualities, we transform the original problem into an unconstrained saddle-point policy optimization problem, and propose an actor-critic algorithm that iteratively and efficiently updates the policy, the Lagrange multiplier, and the Fenchel dual variable. When both the value and policy functions are represented by multi-layer overparameterized neural networks, we prove that our actor-critic algorithm generates a sequence of policies that finds a globally optimal policy at a sublinear rate. We conduct numerical studies using four simulation environments to back up the theoretical results.