Error Bound Analysis for the Regularized Loss of Deep Linear Neural Networks

The optimization foundations of deep linear networks have recently received significant attention. However, because of their inherent nonconvexity and hierarchical structure, analyzing the loss functions of deep linear networks remains a challenging task. In this work, we study the local geometry of the regularized squared loss of deep linear networks around each critical point. Specifically, we obtain a closed-form characterization of the critical point set building on existing results and establish an error bound for the regularized loss under mild conditions on network width and regularization parameters. Notably, this error bound quantifies the distance from a point to the critical point set in terms of the current gradient norm, which can be used to derive linear convergence of first-order methods. To support our theoretical findings, we conduct numerical experiments and demonstrate that gradient descent converges linearly to a critical point when optimizing the regularized loss of deep linear networks.

Funding: R. Jiang is supported by the National Key R&D Program of China [Grant 2023YFA1009300] and the National Natural Science Foundation of China [Grants 72394360 and 72394364]. P. Wang is supported in part by the University of Macau [Grants SRG2025-00043-FST and UMDF-TISF-I/2026/013/FST], in part by the Macau Science and Technology Development Fund (FDCT) [Grant 0091/2025/ITP2], and in part by the National Natural Science Foundation of China (NSFC) [Grant 12601585].