Developing Lagrangian-Based Methods for Nonsmooth Nonconvex Optimization
本文提出一个统一框架，将次梯度方法嵌入拉格朗日方法中，用于求解带非光滑非凸约束的优化问题，并证明其全局收敛性，实验表明在深度学习任务中有效。
In this paper, we consider the minimization of a nonsmooth nonconvex objective function [Formula: see text] over a closed convex subset [Formula: see text] of [Formula: see text], with additional nonsmooth nonconvex constraints [Formula: see text]. We develop a unified framework for developing Lagrangian-based methods, which takes a single-step update to the primal variables by some subgradient methods in each iteration. These subgradient methods are “embedded” into our framework in the sense that they are incorporated as black-box updates to the primal variables. We prove that our proposed framework inherits the global convergence guarantees from these embedded subgradient methods under mild conditions. In addition, we show that our framework can be extended to solve constrained optimization problems with expectation constraints. Based on the proposed framework, we show that a wide range of existing stochastic subgradient methods, including proximal stochastic subgradient descent (SGD), proximal momentum SGD, and proximal adaptive moment estimation method (ADAM), can be embedded into Lagrangian-based methods. Preliminary numerical experiments on deep learning tasks illustrate that our proposed framework yields efficient variants of Lagrangian-based methods with convergence guarantees for nonsmooth nonconvex constrained optimization problems. Funding: The research of X. Hu was supported by the National Natural Science Foundation of China [Grant 12301408]. The research of K.-C. Toh was supported by the Ministry of Education—Singapore [Grant MOE-T2EP20224-0017].