Analytical characterization of sloppiness in neural networks: Insights from linear models
Phys. Rev. E 113, 015306 – Published 21 January, 2026
DOI: https://doi.org/10.1103/fs4g-9nsz
Abstract
Recent experiments have shown that training trajectories of multiple deep neural networks with different architectures, optimization algorithms, hyperparameter settings, and regularization methods evolve on a remarkably low-dimensional “hyperribbon-like” manifold in the space of probability distributions. Inspired by the similarities in the training trajectories of deep networks and linear networks, we analytically characterize this phenomenon for the latter. We show, using tools in dynamical systems theory, that the geometry of this low-dimensional manifold is controlled by (i) the decay rate of the eigenvalues of the input correlation matrix of the training data, (ii) the relative scale of the ground-truth output to the weights at the beginning of training, and (iii) the number of steps of gradient descent. By analytically computing and bounding the contributions of these quantities, we characterize phase boundaries of the region where hyperribbons are to be expected. We also extend our analysis to kernel machines and linear models that are trained with stochastic gradient descent.
Physics Subject Headings (PhySH)
Collections
This article appears in the following collection:
Statistical Physics Meets Machine Learning - Machine Learning Meets Statistical Physics
The Editors of Physical Review E are pleased to present the Collection on Statistical Physics Meets Machine Learning - Machine Learning Meets Statistical Physics, highlighting research at the intersection of machine learning and statistical physics, on the occasion of the two Statistical Physics Meets Machine Learning and the two Machine Learning Meets Statistical Physics sessions at the 2025 Global Physics Summit. The Collection is being guest edited by David Schwab (CUNY, New York) and Yuhai Tu (IBM Watson Research Center Yorktown Heights, NY). Every article published in this collection underwent a rigorous peer review process, adhering to the same high standards applied to all papers. The Physical Review E editorial team managed the peer review and made all editorial decisions.
