Enhancing Representation Power of Deep Neural Networks With Negligible Parameter Growth for Industrial Applications
针对工业场景计算资源有限、数据噪声大的问题，利用常微分方程解释深度残差网络，提出高阶ODE近似方法，以极少额外参数显著提升网络表示能力，在表面缺陷检测等任务上验证了有效性。
In industrial applications where computational resources are finite and data noises are prevalent, the representation power of deep neural networks (DNNs) is crucial. Traditional network structures often require a significant increase in the parameter amount to enhance the representation power, making it difficult to achieve effective representation under parameter amount constraints. In order to alleviate this problem, this work leverages the ordinary differential equation (ODE) interpretation of deep residual networks, elucidating the relationship between the fine-grained connectivity modes of blocks in DNNs and the representation power. We build a bridge from the order of numerical methods and the order of ODEs to the representation power of DNNs. Besides, we show that higher-order ODEs can be approximated by k-step methods incorporating trainable coefficients. Empirically, we validate our theoretical insights by demonstrating the superior representation power of our proposed network structures through enhanced performance on industrial tasks, such as surface defect detection, critical temperature prediction of superconductors, and image classification under noises. The proposed method provides a new approach to the design of network structures for robust and accurate DNNs, enhancing the representation power with a negligible number of additional parameters. The code is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/LongJin-lab/Order-and-Representation-Power</uri>.