Distributed Adaptive Subgradient Algorithms for Online Learning Over Time-Varying Networks
提出一种分布式自适应次梯度算法D-AdaBound，通过裁剪学习率动态调整步长，在凸目标函数下推导出遗憾界，实验证明其优于现有分布式在线学习算法。
Adaptive gradient algorithms have recently become extremely popular because they have been applied successfully in training deep neural networks, such as Adam, AMSGrad, and AdaBound. Despite their success, however, the distributed variant of the adaptive method, which is expected to possess a rapid training speed at the beginning and a good generalization capacity at the end, is rarely studied. To fill the gap, a distributed adaptive subgradient algorithm is presented, called D-AdaBound, where the learning rates are dynamically bounded by clipping the learning rates. Moreover, we obtain the regret bound of D-AdaBound, in which the objective functions are convex. Finally, we confirm the effectiveness of D-AdaBound by simulation experiments on different datasets. The results show the performance improvement of D-AdaBound relative to existing distributed online learning algorithms.