Conditional Gradient Sliding for Convex Optimization
提出一种新的条件梯度滑动算法，通过交替调用线性优化预言机和梯度计算，在保持最优预言机调用次数的同时，大幅减少梯度评估次数，适用于光滑和强凸问题。
In this paper, we present a new conditional gradient type method for convex optimization by calling a linear optimization ($LO$) oracle to minimize a series of linear functions over the feasible set. Different from the classic conditional gradient method, the conditional gradient sliding (CGS) algorithm developed herein can skip the computation of gradients from time to time and, as a result, can achieve the optimal complexity bounds in terms of not only the number of calls to the $LO$ oracle but also the number of gradient evaluations. More specifically, we show that the CGS method requires ${\cal O}(1/\sqrt{\epsilon})$ and ${\cal O}(\log (1/\epsilon))$ gradient evaluations, respectively, for solving smooth and strongly convex problems, while still maintaining the optimal ${\cal O}(1/\epsilon)$ bound on the number of calls to the ${\rm LO}$ oracle. We also develop variants of the CGS method which can achieve the optimal complexity bounds for solving stochastic optimization problems and an important class of saddle point optimization problems. To the best of our knowledge, this is the first time that these types of projection-free optimal first-order methods have been developed in the literature. Some preliminary numerical results have also been provided to demonstrate the advantages of the CGS method.