Decentralized Adaptive TD(λ) Learning With Linear Function Approximation: Nonasymptotic Analysis
提出一种分散式自适应TD(λ)学习算法D-AMSTD(λ)，用线性函数逼近缓解学习率敏感问题，并给出马尔可夫观测下的有限时间性能界，实验表明优于现有分散式TD方法。
In multiagent reinforcement learning, policy evaluation is a central problem. To solve this problem, decentralized temporal-difference (TD) learning is one of the most popular methods, which has been investigated in recent years. However, existing decentralized variants of TD learning often suffer from slow convergence due to the sensitive selection of learning rates. Inspired by the great success of adaptive gradient methods in the training of deep neural networks, this article proposes a decentralized adaptive TD<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(\lambda )$ </tex-math></inline-formula> learning algorithm for general <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\lambda $ </tex-math></inline-formula> with linear function approximation, referred to as <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D-AMSTD</monospace><inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(\boldsymbol {\lambda })$ </tex-math></inline-formula>, which can mitigate the selective sensitivity of learning rates. Furthermore, we establish the finite-time performance bounds of <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D-AMSTD</monospace><inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(\boldsymbol {\lambda })$ </tex-math></inline-formula> under the Markovian observation model. The theoretical results show that <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D-AMSTD</monospace><inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(\boldsymbol {\lambda })$ </tex-math></inline-formula> can linearly converge to an arbitrarily small size of neighborhood of the optimal weight. Finally, we verify the efficacy of <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D-AMSTD</monospace><inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(\boldsymbol {\lambda })$ </tex-math></inline-formula> through a variety of experiments. The results show that <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D-AMSTD</monospace><inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(\boldsymbol {\lambda })$ </tex-math></inline-formula> outperforms existing decentralized TD learning methods.