PfoPG: A Personalized Federated First-Order Policy Gradient Algorithm and Its Nonasymptotic Analysis
针对多智能体强化学习中环境异质性问题，提出了一种个性化联邦一阶策略梯度算法PfoPG，通过自适应混合全局和局部策略来学习个性化模型，并引入基于动量的重要性采样加速收敛，理论证明其收敛率达到O(1/K)，优于现有方法。
This article revisits the federated policy gradient algorithm with environment heterogeneity for finding the optimal policy in multiagent reinforcement learning (RL). Toward this direction, personalized federated RL methods have been presented recently. However, existing personalized federated policy gradient methods may confine the personalized capacity of local policy models. In order to tackle this challenge, this article develops a provably convergent personalized federated first-order policy gradient algorithm, referred to as PfoPG, which learns a personalized policy model by adaptively mixing optimal global and local policies. Moreover, the momentum-based importance sampling is also introduced into PfoPG to improve its convergence speed. Meanwhile, this article rigorously analyzes the nonasymptotic convergence behavior of PfoPG. More specifically, PfoPG converges to a stationary policy with rate <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O</i>(1/<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</i>), where <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</i> denotes the number of iterations. Compared to the state-of-the-art federated policy gradient methods, PfoPG can improve the convergence rate from <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O</i>(1/<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</i><sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2/3</sup>) to <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O</i>(1/<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</i>). Finally, we verify the effectiveness of PfoPG by various experiments based on the multiagent particle environment.