- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Universal scaling laws of absorbing phase transitions in artificial deep neural networks

Keiichi Tamai1,*, Tsuyoshi Okubo1, Truong Vinh Truong Duy2, Naotake Natori2, and Synge Todo3,1,4

  • *Contact author: tamai@phys.s.u-tokyo.ac.jp
PDF

Phys. Rev. Research 7, 033072 – Published 18 July, 2025

DOI: https://doi.org/10.1103/jp61-6sp2

Abstract

We demonstrate that conventional artificial deep neural networks operating near the phase boundary of the signal propagation dynamics—also known as the edge of chaos—exhibit universal scaling laws of absorbing phase transitions in nonequilibrium statistical mechanics. We exploit the fully deterministic nature of the propagation dynamics to elucidate an analogy between a signal collapse in the neural networks and an absorbing state (a state that the system can enter but cannot escape from). Our numerical results indicate that the multilayer perceptrons and the convolutional neural networks belong to the mean-field and the directed percolation universality classes, respectively. Also, the finite-size scaling is successfully applied, suggesting a potential connection to the depth-width trade-off in deep learning. Furthermore, our analysis of the training dynamics under the gradient descent reveals that hyperparameter tuning to the phase boundary is necessary but insufficient for achieving optimal generalization in deep networks. Remarkably, nonuniversal metric factors associated with the scaling laws are shown to play a significant role in concretizing the above observations. These findings highlight the usefulness of the notion of criticality for analyzing the behavior of artificial deep neural networks and offer new insights toward a unified understanding of the essential relationship between criticality and intelligence.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (91)

  1. A. M. Turing, I.—Computing machinery and intelligence, Mind LIX, 433 (1950).
  2. C. G. Langton, Computation at the edge of chaos: Phase transitions and emergent computation, Physica D 42, 12 (1990).
  3. M. A. Muñoz, Colloquium: Criticality and dynamical scaling in living systems, Rev. Mod. Phys. 90, 031001 (2018).
  4. J. Bickle, Reducing mind to molecular pathways: Explicating the reductionism implicit in current cellular and molecular neuroscience, Synthese 151, 411 (2006).
  5. E. R. Kandel, J. D. Koester, S. H. Mack, and S. A. Siegelbaum, Principles of Neural Science, Sixth Edition (McGraw–Hill, New York, 2021).
  6. J. M. Beggs, Addressing skepticism of the critical brain hypothesis, Front. Comput. Neurosci. 16, 703865 (2022).
  7. M. Girardi-Schappo, Brain criticality beyond avalanches: open problems and how to approach them, J. Phys. Complex. 2, 031003 (2021).
  8. C. Gros, A devil's advocate view on 'self-organized' brain criticality, J. Phys. Complex. 2, 031001 (2021).
  9. H. Hinrichsen, Non-equilibrium critical phenomena and phase transitions into absorbing states, Adv. Phys. 49, 815 (2000).
  10. M. Henkel, H. Hinrichsen, and S. Lübeck, Non-Equilibrium Phase Transitions, 1st ed. (Springer, Berlin, 2008).
  11. R. Dickman, A. Vespignani, and S. Zapperi, Self-organized criticality as an absorbing-state phase transition, Phys. Rev. E 57, 5095 (1998).
  12. P. Bak, C. Tang, and K. Wiesenfeld, Self-organized criticality: An explanation of the 1/f noise, Phys. Rev. Lett. 59, 381 (1987).
  13. J. Hesse and T. Gross, Self-organized criticality as a fundamental property of neural systems, Front. Syst. Neurosci. 8, 166 (2014).
  14. D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, Mastering the game of go with deep neural networks and tree search, Nature (London) 529, 484 (2016).
  15. F. Stahlberg, Neural machine translation: A review, J. Artif. Intell. Res. 69, 343 (2020).
  16. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, High-resolution image synthesis with latent diffusion models, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'22) (IEEE, Los Alamitos, CA, 2022), pp. 10684–10695.
  17. B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli, Exponential expressivity in deep neural networks through transient chaos, in Advances in Neural Information Processing Systems, edited by D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Curran Associates, Inc., Red Hook, NY, 2016), Vol. 29.
  18. S. S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein, Deep information propagation, in The Fifth International Conference on Learning Representations (2017).
  19. L. Xiao, J. Pennington, and S. Schoenholz, Disentangling trainability and generalization in deep neural networks, in Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, edited by H. D. III and A. Singh (Journal of Machine Learning Research Inc., Cambridge, MA, 2020), Vol. 119, pp. 10462–10472.
  20. S. Peluchetti and S. Favaro, Infinitely deep neural networks as diffusion processes, in Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research Vol. 108, edited by S. Chiappa and R. Calandra (Journal of Machine Learning Research Inc., Cambridge, MA, 2020), pp. 1126–1136.
  21. S. Hayou, A. Doucet, and J. Rousseau, The curse of depth in kernel regime, in Proceedings on "I (Still) Can't Believe It's Not Better!” at NeurIPS 2021 Workshops, Proceedings of Machine Learning Research Vol. 163, edited by M. F. Pradier, A. Schein, S. Hyland, F. J. R. Ruiz, and J. Z. Forde (Journal of Machine Learning Research Inc., Cambridge, MA, 2022), pp. 41–47.
  22. S. Hayou, A. Doucet, and J. Rousseau, On the impact of the activation function on deep neural networks training, in Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research Vol. 97, edited by K. Chaudhuri and R. Salakhutdinov (Journal of Machine Learning Research Inc., Cambridge, MA, 2019), pp. 2672–2680.
  23. D. A. Roberts, S. Yaida, and B. Hanin, The Principles of Deep Learning Theory (Cambridge University Press, Cambridge, UK, 2022).
  24. L. Xiao, Y. Bahri, J. Sohl-Dickstein, S. Schoenholz, and J. Pennington, Dynamical isometry and a mean field theory of CNNs: How to train 10,000-layer vanilla convolutional neural networks, in Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research Vol. 80, edited by J. Dy and A. Krause (Journal of Machine Learning Research Inc., Cambridge, MA, 2018), pp. 5393–5402.
  25. G. Yang and S. Schoenholz, Mean field residual networks: On the edge of chaos, in Advances in Neural Information Processing Systems edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Curran Associates, Inc., Red Hook, NY, 2017), Vol. 30.
  26. K. T. Grosvenor and R. Jefferson, The edge of chaos: Quantum field theory and deep neural networks, SciPost Phys. 12, 081 (2022).
  27. S. Yaida, Non-Gaussian processes and neural networks at finite widths, in Proceedings of The First Mathematical and Scientific Machine Learning Conference, Proceedings of Machine Learning Research Vol. 107, edited by J. Lu and R. Ward (Journal of Machine Learning Research Inc., Cambridge, MA, 2020), pp. 165–192.
  28. A. E. Ferdinand and M. E. Fisher, Bounded and inhomogeneous ising models. I. Specific-heat anomaly of a finite lattice, Phys. Rev. 185, 832 (1969).
  29. H. K. Janssen, B. Schaub, and B. Schmittmann, New universal short-time scaling behaviour of critical relaxation processes, Z. Phys. B 73, 539 (1989).
  30. A. Jacot, F. Gabriel, and C. Hongler, Neural tangent kernel: Convergence and generalization in neural networks, in Advances in Neural Information Processing Systems, edited by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Curran Associates, Inc., Red Hook, NY, 2018), Vol. 31.
  31. J. Sohl-Dickstein, R. Novak, S. S. Schoenholz, and J. Lee, On the infinite width limit of neural networks with a standard parameterization, arXiv:2001.07301.
  32. J. Lee, L. Xiao, S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Pennington, Wide neural networks of any depth evolve as linear models under gradient descent, in Advances in Neural Information Processing Systems, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Curran Associates, Inc., Red Hook, NY, 2019), Vol. 32.
  33. The standard Lagrange's (prime) notation for differentiation is used throughout the paper.
  34. R. M. Neal, Priors for infinite networks, in Bayesian Learning for Neural Networks (Springer New York, New York, 1996), pp. 29–53.
  35. C. Williams, Computing with infinite networks, in Advances in Neural Information Processing Systems, edited by M. Mozer, M. Jordan, and T. Petsche (MIT Press, Cambridge, MA, 1996), Vol. 9.
  36. J. Lee, J. Sohl-dickstein, J. Pennington, R. Novak, S. Schoenholz, and Y. Bahri, Deep neural networks as Gaussian processes, in The Sixth International Conference on Learning Representations (2018).
  37. A. G. de G. Matthews, J. Hron, M. Rowland, R. E. Turner, and Z. Ghahramani, Gaussian process behaviour in wide deep neural networks, in The Sixth International Conference on Learning Representations (2018).
  38. V. Privman and M. E. Fisher, Universal critical amplitudes in finite-size scaling, Phys. Rev. B 30, 322 (1984).
  39. In particular, if we normalize the input so that q(1) matches the fixed point q* of the mean-field theory (7), then ω equals the proportionality constant between the input cosine distance ρ(0) and the initial condition ρ(1) of the mean-field theory: ρ(1)=ωρ(0) with ω=1−σb2/q*.
  40. N. Dehak, P. J. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, Front-end factor analysis for speaker verification, IEEE Trans. Aud. Speech Lang. Process. 19, 788 (2011).
  41. Y. Cao and Q. Gu, Generalization bounds of stochastic gradient descent for wide and deep neural networks, in Advances in Neural Information Processing Systems, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Curran Associates, Inc., Red Hook, NY, 2019), Vol. 32.
  42. S. Arora, S. S. Du, W. Hu, Z. Li, R. R. Salakhutdinov, and R. Wang, On exact computation with an infinitely wide neural net, in Advances in Neural Information Processing Systems, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Curran Associates, Inc., Red Hook, NY, 2019), Vol. 32.
  43. This peculiarity explains why the finite size scaling in the multilayer perceptrons is different from that in the contact process [91] on a complete graph, where one finds the same β and ν∥ [Eq. (22)] but the exponent for finite-size scaling (45) is replaced with −1/2. If the third-order cumulant remained non-zero, then the leading order for the correction would be an order of n−12.
  44. J. E. Kolassa, Edgeworth series, in Series Approximation Methods in Statistics (Springer New York, New York, 2006), pp. 31–62.
  45. Y. Bahri, B. Hanin, A. Brossollet, V. Erba, C. Keup, R. Pacelli, and J. B. Simon, Les Houches lectures on deep learning at large and infinite width, J. Stat. Mech. (2024) 104012.
  46. B. Hanin, Random fully connected neural networks as perturbatively solvable hierarchies, J. Mach. Learn. Res. 25, 1 (2024).
  47. J. Huang and H.-T. Yau, Dynamics of deep neural networks and neural tangent hierarchy, in Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research Vol. 119, edited by H. Daumé III and A. Singh (Journal of Machine Learning Research Inc., Cambridge, MA, 2020), pp. 4542–4551.
  48. A. Garriga-Alonso, C. E. Rasmussen, and L. Aitchison, Deep convolutional networks as shallow gaussian processes, in The Seventh International Conference on Learning Representations (2019).
  49. R. Coutinho and B. Fernandez, Extended symbolic dynamics in bistable CML: Existence and stability of fronts, Physica D 108, 60 (1997).
  50. B. Fernandez and L. Raymond, Propagating fronts in a bistable coupled map lattice, J. Stat. Phys. 86, 337 (1997).
  51. K. Kaneko, Period-doubling of kink-antikink patterns, quasiperiodicity in antiferro-like structures and spatial intermittency in coupled logistic lattice: Towards a prelude of a "field theory of chaos,” Prog. Theor. Phys. 72, 480 (1984).
  52. G. Grinstein and M. A. Muñoz, The statistical mechanics of absorbing states, in Fourth Granada Lectures in Computational Physics, edited by P. L. Garrido and J. Marro (Springer, Berlin, 1997), pp. 223–270.
  53. S. R. Broadbent and J. M. Hammersley, Percolation processes: I. Crystals and mazes, Math. Proc. Cambr. Philos. Soc. 53, 629 (1957).
  54. I. Jensen, Low-density series expansions for directed percolation: I. A new efficient algorithm with applications to the square lattice, J. Phys. A: Math. Gen. 32, 5233 (1999).
  55. Other possible (albeit relatively minor) reasons for the deviation from the scaling collapse include the finite-size effect and deviation of the noise from the asymptotic behavior (which is particularly true if the number c of channels is small).
  56. C. A. Voigt and R. M. Ziff, Epidemic analysis of the second-order transition in the Ziff-Gulari-Barshad surface-reaction model, Phys. Rev. E 56, R6241(R) (1997).
  57. J. Wang, Z. Zhou, Q. Liu, T. M. Garoni, and Y. Deng, High-precision Monte Carlo study of directed percolation in (d+1) dimensions, Phys. Rev. E 88, 042102 (2013).
  58. A. Guttmann, Indicators of solvability for lattice models, Discr. Math. 217, 167 (2000).
  59. Z. Chen, Y. Cao, Q. Gu, and T. Zhang, A generalized neural tangent kernel analysis for two-layer neural networks, in Advances in Neural Information Processing Systems, edited by H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Curran Associates, Inc., Red Hook, NY, 2020), Vol. 33, pp. 13363–13373.
  60. P. Ju, X. Lin, and N. Shroff, On the generalization power of overfitted two-layer neural tangent kernel models, in Proceedings of the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research, edited by M. Meila and T. Zhang (Journal of Machine Learning Research Inc., Cambridge, MA, 2021), Vol. 139, pp. 5137–5147.
  61. T. Yu and H. Zhu, Hyper-parameter optimization: A review of algorithms and applications, arXiv:2003.05689.
  62. L. Chizat, E. Oyallon, and F. Bach, On lazy training in differentiable programming, in Advances in Neural Information Processing Systems, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Curran Associates, Inc., Red Hook, NY, 2019), Vol. 32.
  63. E. Nichani, Y. Bai, and J. D. Lee, Identifying good directions to escape the NTK regime and efficiently learn low-degree plus sparse polynomials, in Advances in Neural Information Processing Systems, edited by S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Curran Associates, Inc., Red Hook, NY, 2022), Vol. 35, pp. 14568–14581.
  64. C. Gervais, L.-P. Boucher, G. M. Villar, U. Lee, and C. Duclos, A scoping review for building a criticality-based conceptual framework of altered states of consciousness, Front. Syst. Neurosci. 17, 1085902 (2023).
  65. Y. Xu, A. Schneider, R. Wessel, and K. B. Hengen, Sleep restores an optimal computational regime in cortical networks, Nat. Neurosci. 27, 328 (2024).
  66. N. Friedman, S. Ito, B. A. W. Brinkman, M. Shimono, R. E. L. DeVille, K. A. Dahmen, J. M. Beggs, and T. C. Butler, Universal critical dynamics in high resolution neuronal avalanche data, Phys. Rev. Lett. 108, 208102 (2012).
  67. W. L. Shew, W. P. Clawson, J. Pobst, Y. Karimipanah, N. C. Wright, and R. Wessel, Adaptation to sensory input tunes visual cortex to criticality, Nat. Phys. 11, 659 (2015).
  68. E. K. Kosmidis, Y. F. Contoyiannis, C. Papatheodoropoulos, and F. K. Diakonos, Traits of criticality in membrane potential fluctuations of pyramidal neurons in the CA1 region of rat hippocampus, Eur. J. Neurosci. 48, 2343 (2018).
  69. A. J. Fontenele, N. A. P. de Vasconcelos, T. Feliciano, L. A. A. Aguiar, C. Soares-Cunha, B. Coimbra, L. Dalla Porta, S. Ribeiro, A. J. a. Rodrigues, N. Sousa, P. V. Carelli, and M. Copelli, Criticality between Cortical States, Phys. Rev. Lett. 122, 208101 (2019).
  70. B. Zheng, Monte Carlo simulations and numerical solutions of short-time critical dynamics, Physica A 283, 80 (2000).
  71. U. Hasson, J. Chen, and C. J. Honey, Hierarchical process memory: Memory as an integral component of information processing, Trends Cogn. Sci. 19, 304 (2015).
  72. M. Golesorkhi, J. Gomez-Pilar, F. Zilio, N. Berberian, A. Wolff, M. C. E. Yagoub, and G. Northoff, The brain and its time: Intrinsic neural timescales are key for input processing, Commun. Biol. 4, 970 (2021).
  73. Namely, (a) derivation of more precise design principles of the neural networks from the universal scaling laws, possibly with prior knowledge about a dataset at hand (Sec. 4b) and (b) uncovering the details of the trade-off between the width and the depth (Sec. 4c).
  74. N. Bertschinger, T. Natschläger, and R. Legenstein, At the edge of chaos: Real-time computations and self-organized criticality in recurrent neural networks, in Advances in Neural Information Processing Systems, edited by L. Saul, Y. Weiss, and L. Bottou (MIT Press, Cambridge, MA, 2004), Vol. 17.
  75. S. Takasu and T. Aoyagi, Suppression of chaos in a partially driven recurrent neural network, Phys. Rev. Res. 6, 013172 (2024).
  76. K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning for image recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'16) (IEEE, Los Alamitos, CA, 2016), pp. 770–778.
  77. G. Lan, P. Sartori, S. Neumann, V. Sourjik, and Y. Tu, The energy–speed–accuracy trade-off in sensory adaptation, Nat. Phys. 8, 422 (2012).
  78. R. Rao and L. Peliti, Thermodynamics of accuracy in kinetic proofreading: Dissipation and efficiency trade-offs, J. Stat. Mech. (2015) P06001.
  79. K. Banerjee, A. B. Kolomeisky, and O. A. Igoshin, Elucidating interplay of speed and accuracy in biological error correction, Proc. Natl. Acad. Sci. USA 114, 5183 (2017).
  80. T. E. Ouldridge, C. C. Govern, and P. R. ten Wolde, Thermodynamics of computational copying in biochemical systems, Phys. Rev. X 7, 021004 (2017).
  81. K. Ikeda, T. Uda, D. Okanohara, and S. Ito, Speed-accuracy trade-off for the diffusion models: Wisdom from nonequilibrium thermodynamics and optimal transport, arXiv:2407.04495 [Phys. Rev. X (to be published)].
  82. S. Zeraati, F. H. Jafarpour, and H. Hinrichsen, Entropy production of nonequilibrium steady states with irreversible transitions, J. Stat. Mech. (2012) L12001.
  83. K. Harada and N. Kawashima, Entropy governed by the absorbing state of directed percolation, Phys. Rev. Lett. 123, 090601 (2019).
  84. https://qsw.phys.s.u-tokyo.ac.jp/.
  85. K. He, X. Zhang, S. Ren, and J. Sun, Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification, in Proceedings of the IEEE International Conference on Computer Vision (ICCV'15) (IEEE, Los Alamitos, CA, 2015).
  86. Y. Cho and L. Saul, Kernel methods for deep learning, in Advances in Neural Information Processing Systems, edited by Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta (Curran Associates, Inc., Red Hook, NY, 2009), Vol. 22.
  87. L. Lu, Y. Shin, Y. Su, and G. Em Karniadakis, Dying ReLU and initialization: Theory and numerical examples, Commun. Comput. Phys. 28, 1671 (2020).
  88. A. L. Maas, A. Y. Hannun, and A. Y. Ng, Rectifier nonlinearities improve neural network acoustic models, in ICML Workshop on Deep Learning for Audio, Speech, and Language Processing (WDLASL'13) (2013).
  89. S. R. Dubey, S. K. Singh, and B. B. Chaudhuri, Activation functions in deep learning: A comprehensive survey and benchmark, Neurocomputing 503, 92 (2022).
  90. B. Xu, N. Wang, T. Chen, and M. Li, Empirical evaluation of rectified activations in convolutional network, arXiv:1505.00853.
  91. T. E. Harris, Contact interactions on a lattice, Ann. Probab. 2, 969 (1974).

Outline

Information

Phys. Rev. Research 7, 033072– Published 18 July, 2025
  • Received 30 October 2024
  • Accepted 10 June 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation