Greedy Criterion in Orthogonal Greedy Learning
本文发现正交贪婪学习中的最速梯度下降并非唯一贪婪准则，提出一种新的“-贪婪阈值”准则，并据此导出简单终止规则，在保持几乎最优学习率的同时降低计算量。
Orthogonal greedy learning (OGL) is a stepwise learning scheme that starts with selecting a new atom from a specified dictionary via the steepest gradient descent (SGD) and then builds the estimator through orthogonal projection. In this paper, we found that SGD is not the unique greedy criterion and introduced a new greedy criterion, called as " -greedy threshold" for learning. Based on this new greedy criterion, we derived a straightforward termination rule for OGL. Our theoretical study shows that the new learning scheme can achieve the existing (almost) optimal learning rate of OGL. Numerical experiments are also provided to support that this new scheme can achieve almost optimal generalization performance while requiring less computation than OGL.