- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Letters

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Editors' Suggestion
  • Open Access

Spring-Block Theory of Feature Learning in Deep Neural Networks

Cheng Shi1, Liming Pan2, and Ivan Dokmanić1,3,*

  • *Contact author: ivan.dokmanic@unibas.ch
PDF

Phys. Rev. Lett. 134, 257301 – Published 27 June, 2025

DOI: https://doi.org/10.1103/ys4n-2tj3

Abstract

Feature-learning deep nets progressively collapse data to a regular low-dimensional geometry. How this emerges from the collective action of nonlinearity, noise, learning rate, and other factors, has eluded first-principles theories built from microscopic neuronal dynamics. We exhibit a noise–nonlinearity phase diagram that identifies regimes where shallow or deep layers learn more effectively and propose a macroscopic mechanical theory that reproduces the diagram and links feature learning across layers to generalization.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (65)

  1. M. D. Zeiler and R. Fergus, Visualizing and understanding convolutional networks, in Proceedings of the Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland 2014, Part I 13 (Springer, New York, 2014), pp. 818–833.
  2. V. Papyan, X. Han, and D. L. Donoho, Prevalence of neural collapse during the terminal phase of deep learning training, Proc. Natl. Acad. Sci. U.S.A. 117, 24652 (2020).
  3. H. He and W. J. Su, A law of data separation in deep learning, Proc. Natl. Acad. Sci. U.S.A. 120, e2221704120 (2023).
  4. J. Zarka, F. Guth, and S. Mallat, Separation and concentration in deep networks, in Proceedings of the ICLR 2021-9th International Conference on Learning Representations (OpenReview.net, Vienna, Austria, 2021).
  5. A. Rangamani, M. Lindegaard, T. Galanti, and T. A. Poggio, Feature learning in deep classifiers through intermediate neural collapse, in Proceedings of the International Conference on Machine Learning (PMLR, Honolulu, 2023), pp. 28729–28745.
  6. V. Kothapalli, T. Tirer, and J. Bruna, A neural collapse perspective on feature evolution in graph neural networks, Adv. Neural Inf. Process. Syst. 36, 14134 (2023).
  7. S. Qin, N. Mudur, and C. Pehlevan, Contrastive similarity matching for supervised learning, Neural Comput. 33, 1300 (2021).
  8. T. Hofmann, B. Schölkopf, and A. J. Smola, Kernel methods in machine learning, Ann. Stat. 36, 1171 (2008).
  9. A. Rahimi and B. Recht, Random features for large-scale kernel machines, Adv. Neural Inf. Process. Syst. 20, 1177 (2007).
  10. A. Daniely, R. Frostig, and Y. Singer, Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity, Adv. Neural Inf. Process. Syst. 29, 2261 (2016).
  11. A. Radhakrishnan, D. Beaglehole, P. Pandit, and M. Belkin, Mechanism for feature learning in neural networks and backpropagation-free machine learning models, Science 383, 1461 (2024).
  12. Y. Lou, C. E. Mingard, and S. Hayou, Feature learning and signal propagation in deep neural networks, in Proceedings of the International Conference on Machine Learning (PMLR, Baltimore, 2022), pp. 14248–14282.
  13. A. Baratin, T. George, C. Laurent, R. D. Hjelm, G. Lajoie, P. Vincent, and S. Lacoste-Julien, Implicit regularization via neural feature alignment, in Proceedings of the International Conference on Artificial Intelligence and Statistics (PMLR, 2021), pp. 2269–2277.
  14. G. Yang and E. J. Hu, Tensor programs IV: Feature learning in infinite-width neural networks, in Proceedings of the International Conference on Machine Learning (PMLR, 2021), pp. 11727–11737.
  15. L. Chizat and P. Netrapalli, Steering deep feature learning with backward aligned feature updates, arXiv:2311.18718.
  16. S. Mei, A. Montanari, and P.-M. Nguyen, A mean field view of the landscape of two-layer neural networks, Proc. Natl. Acad. Sci. U.S.A. 115, E7665 (2018).
  17. L. Chizat and F. Bach, On the global convergence of gradient descent for over-parametrized models using optimal transport, Adv. Neural Inf. Process. Syst. 31, 3040 (2018).
  18. Y. Wang, J. Lacotte, and M. Pilanci, The hidden convex optimization landscape of regularized two-layer relu networks: An exact characterization of optimal solutions, in Proceedings of the International Conference on Learning Representations (OpenReview.net, Vienna, Austria, 2021).
  19. L. Arnaboldi, L. Stephan, F. Krzakala, and B. Loureiro, From high-dimensional & mean-field dynamics to dimensionless odes: A unifying approach to sgd in two-layers networks, in Proceedings of the Thirty Sixth Annual Conference on Learning Theory (PMLR, Bangalore, 2023), pp. 1199–1227.
  20. T. Luo, Z.-Q. J. Xu, Z. Ma, and Y. Zhang, Phase diagram for two-layer relu neural networks at infinite-width limit, J. Mach. Learn. Res. 22, 1 (2021).
  21. A. Maillard, A. S. Bandeira, D. Belius, I. Dokmanić, and S. Nakajima, Injectivity of relu networks: Perspectives from statistical physics, arXiv:2302.14112.
  22. R. Pacelli, S. Ariosto, M. Pastore, F. Ginelli, M. Gherardi, and P. Rotondo, A statistical mechanics framework for bayesian deep neural networks beyond the infinite-width limit, Nat. Mach. Intell. 5, 1497 (2023).
  23. P. Baglioni, R. Pacelli, R. Aiudi, F. Di Renzo, A. Vezzani, R. Burioni, and P. Rotondo, Predictive power of a bayesian effective action for fully connected one hidden layer neural networks in the proportional limit, Phys. Rev. Lett. 133, 027301 (2024).
  24. H. Cui, L. Pesce, Y. Dandi, F. Krzakala, Y. M. Lu, L. Zdeborová, and B. Loureiro, Asymptotics of feature learning in two-layer networks after one gradient-step, arXiv preprint arXiv:2402.04980 (2024).
  25. J. Sohl-Dickstein, The boundary of neural network trainability is fractal, arXiv preprint arXiv:2402.06184 (2024).
  26. R. Marino and F. Ricci-Tersenghi, Phase transitions in the mini-batch size for sparse and dense two-layer neural networks, Mach. Learn. 5, 015015 (2024).
  27. Y. Dandi, E. Troiani, L. Arnaboldi, L. Pesce, L. Zdeborová, and F. Krzakala, The benefits of reusing batches for gradient descent in two-layer networks: Breaking the curse of information and leap exponents, arXiv preprint arXiv:2402.03220 (2024).
  28. S. Ciceri, L. Cassani, M. Osella, P. Rotondo, F. Valle, and M. Gherardi, Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalization, Nat. Mach. Intell. 6, 40 (2024).
  29. E. Boursier and N. Flammarion, Early alignment in two-layer networks training is a two-edged sword, arXiv:2401.10791.
  30. S. Goldt, M. Mézard, F. Krzakala, and L. Zdeborová, Modeling the influence of data structure on learning in neural networks: The hidden manifold model, Phys. Rev. X 10, 041044 (2020).
  31. A. Jacot, F. Gabriel, and C. Hongler, Neural tangent kernel: Convergence and generalization in neural networks, Adv. Neural Inf. Process. Syst. 31, 8580 (2018).
  32. F. Guth, B. Ménard, G. Rochette, and S. Mallat, A rainbow in deep network black boxes, arXiv:2305.18512.
  33. K. H. R. Chan, Y. Yu, C. You, H. Qi, J. Wright, and Y. Ma, ReduNet: A white-box deep network from the principle of maximizing rate reduction, J. Mach. Learn. Res. 23, 1 (2022).
  34. C. Fang, H. He, Q. Long, and W. J. Su, Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training, Proc. Natl. Acad. Sci. U.S.A. 118, e2103091118 (2021).
  35. S. Arora, N. Cohen, N. Golowich, and W. Hu, A convergence analysis of gradient descent for deep linear neural networks, arXiv:1810.02281.
  36. Q. Li and H. Sompolinsky, Statistical mechanics of deep linear neural networks: The backpropagating kernel renormalization, Phys. Rev. X 11, 031059 (2021).
  37. B. Bordelon and C. Pehlevan, Self-consistent dynamical field theory of kernel evolution in wide neural networks, Adv. Neural Inf. Process. Syst. 35, 32240 (2022).
  38. B. Bordelon, A. Atanasov, and C. Pehlevan, A dynamical model of neural scaling laws, arXiv:2402.01092.
  39. G. Ben Arous, R. Gheissari, and A. Jagannath, High-dimensional limit theorems for SGD: Effective dynamics and critical scaling, Adv. Neural Inf. Process. Syst. 35, 25349 (2022).
  40. G. B. Arous, R. Gheissari, J. Huang, and A. Jagannath, High-dimensional SGD aligns with emerging outlier eigenspaces, arXiv:2310.03010.
  41. C. Yaras, P. Wang, W. Hu, Z. Zhu, L. Balzano, and Q. Qu, The law of parsimony in gradient descent for learning deep linear networks, arXiv:2306.01154.
  42. P. Wang, X. Li, C. Yaras, Z. Zhu, L. Balzano, W. Hu, and Q. Qu, Understanding deep representation learning via layerwise feature compression and discrimination, arXiv:2311.02960.
  43. E. Gardner and B. Derrida, Optimal storage properties of neural network models, J. Phys. A 21, 271 (1988).
  44. C. Baldassi, C. Lauditi, E. M. Malatesta, G. Perugini, and R. Zecchina, Unveiling the structure of wide flat minima in neural networks, Phys. Rev. Lett. 127, 278301 (2021).
  45. S. Becker, Y. Zhang, and A. A. Lee, Geometry of energy landscapes and the optimizability of deep neural networks, Phys. Rev. Lett. 124, 108301 (2020).
  46. C. Shi, L. Pan, H. Hu, and I. Dokmanić, Homophily modulates double descent generalization in graph convolution networks, Proc. Natl. Acad. Sci. U.S.A. 121, e2309504121 (2024).
  47. J. Pennington and Y. Bahri, Geometry of neural network loss surfaces via random matrix theory, in Proceedigs of the International conference on machine learning (PMLR, Sydney, 2017), pp. 2798–2806.
  48. X. Fernández-Real and A. Figalli, The continuous formulation of shallow neural networks as wasserstein-type gradient flows, in Proceedings of the Analysis at Large: Dedicated to the Life and Work of Jean Bourgain (Springer, New York, 2022), pp. 29–57.
  49. N. Yang, C. Tang, and Y. Tu, Stochastic gradient descent introduces an effective landscape-dependent regularization favoring flat solutions, Phys. Rev. Lett. 130, 237101 (2023).
  50. A. Sclocchi and M. Wyart, On the different regimes of stochastic gradient descent, Proc. Natl. Acad. Sci. U.S.A. 121, e2316301121 (2024).
  51. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, Dropout: A simple way to prevent neural networks from overfitting, J. Mach. Lean. Res. 15, 1929 (2014).
  52. S. Sukhbaatar and R. Fergus, Learning from noisy labels with deep neural networks, arXiv:1406.2080.
  53. See Supplemental Material at http://link.aps.org/supplemental/10.1103/ys4n-2tj3, which includes: (1) a folding ruler experiment, (2) additional experimental results on various networks and datasets, (3) details of numerical experiments and reproducibility, (4) some other metrics for data separation of intermediate features, (5) regression, (6) additional details about the spring block systems and (7) scaling and quantity of noise sources.
  54. K. He, X. Zhang, S. Ren, and J. Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, in Proceedings of the IEEE international conference on computer vision (IEEE Computer Society, Santiago, Chile, 2015), pp. 1026–1034.
  55. B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli, Exponential expressivity in deep neural networks through transient chaos, Adv. Neural Inf. Process. Syst. 29, 3368 (2016).
  56. S. S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein, Deep information propagation, arXiv:1611.01232.
  57. S. Ioffe and C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, in Proceedings of the International Conference on Machine Learning (PMLR, Lille, 448–456.
  58. H. Hu and Y. M. Lu, Universality laws for high-dimensional learning with random features, IEEE Trans. Theory 69, 1932 (2022).

  59. He and Su [3] and some other previous works [2, 4] define the data separation slightly differently as D^ℓ≔log[Tr(ΣℓwΣℓb+)], where Σ+ denotes the Moore-Penrose inverse of Σ. As mentioned in [42], Eq. (3) can be regarded as a simplified version of the above definition.

  60. S. Arora, N. Cohen, and E. Hazan, On the optimization of deep networks: Implicit acceleration by overparametrization, in International conference on machine learning (PMLR, Stockholm, 2018), pp. 244–253.
  61. C. H. Martin and M. W. Mahoney, Rethinking generalization requires revisiting old ideas: Statistical mechanics approaches and complex learning behavior, arXiv:1710.09553.
  62. Y. Zhou, T. Pang, K. Liu, M. W. Mahoney, Y. Yang et al., Temperature balancing, layer-wise weight analysis, and neural network training, Adv. Neural Inf. Process. Syst. 36, 63542 (2024).
  63. S. Shi, D. Guo, and J. Luo, Micro/atomic-scale vibration induced superlubricity, Frict. Wear Mach. 9, 1163 (2021).
  64. R. Burridge and L. Knopoff, Model and theoretical seismicity, Bull. Seismol. Soc. Am. 57, 341 (1967).
  65. J. J. Hopfield, Neurons, dynamics and computation, Phys. Today 47, No. 2, 40 (1994).

Outline

Information

Phys. Rev. Lett. 134, 257301– Published 27 June, 2025
  • Received 25 October 2024
  • Accepted 29 May 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation