- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Dynamic neuron approach to deep neural networks: Decoupling neurons for renormalization group analysis

Donghee Lee*, Hye-Sung Lee†, and Jaeok Yi‡

  • *Contact author: dhlee641@kaist.ac.kr
  • †Contact author: hyesung.lee@kaist.ac.kr
  • ‡Contact author: wodhr1541@kaist.ac.kr
PDF

Phys. Rev. Research 7, 023276 – Published 17 June, 2025

DOI: https://doi.org/10.1103/tf5z-7q29

Abstract

Deep neural network architectures often consist of repetitive structural elements. We introduce an approach that reveals these patterns and can be broadly applied to the study of deep learning. Similarly to how a power strip helps untangle and organize complex cable connections, this approach treats neurons as additional degrees of freedom in interactions, simplifying the structure and enhancing the intuitive understanding of interactions within deep neural networks. Furthermore, it reveals the translational symmetry of deep neural networks, which simplifies the application of the renormalization group transformation, a method that effectively analyzes the scaling behavior of the system. By utilizing translational symmetry and renormalization group transformations, we can analyze critical phenomena. This approach may open new avenues for studying deep neural networks using statistical physics.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (75)

  1. F.-L. Fan, J. Xiong, M. Li, and G. Wang, On interpretability of artificial neural networks: A survey, IEEE Trans. Radiat. Plasma Med. Sci. 5, 741 (2021).
  2. X. Li, H. Xiong, X. Li, X. Wu, X. Zhang, J. Liu, J. Bian, and D. Dou, Interpretable deep learning: Interpretation, interpretability, trustworthiness, and beyond, Knowl. Inf. Syst. 64, 3197 (2022).
  3. Y. Bahri, J. Kadmon, J. Pennington, S. S. Schoenholz, J. Sohl-Dickstein, and S. Ganguli, Statistical mechanics of deep learning, Annu. Rev. Condens. Matter Phys. 11, 501 (2020).
  4. P. Carnevali and S. Patarnello, Exhaustive thermodynamical analysis of Boolean learning networks, Europhys. Lett. 4, 1199 (1987).
  5. N. Tishby, E. Levin, and S. A. Solla, Consistent inference of probabilities in layered networks: Predictions and generalizations, in Proceedings IEEE International Conference on Neural Networks (IEEE, Piscataway, NJ, 1989), pp. 403–409.
  6. H. Sompolinsky, N. Tishby, and H. S. Seung, Learning from examples in large neural networks, Phys. Rev. Lett. 65, 1683 (1990).
  7. E. Levin, N. Tishby, and S. A. Solla, A statistical approach to learning and generalization in layered neural networks, Proc. IEEE 78, 1568 (1990).
  8. H. S. Seung, H. Sompolinsky, and N. Tishby, Statistical mechanics of learning from examples, Phys. Rev. A 45, 6056 (1992).
  9. A. Engel, Statistical Mechanics of Learning (Cambridge University Press, Cambridge, 2001).
  10. M. Mezard and A. Montanari, Information, Physics, and Computation (Oxford University Press, Oxford, 2009).
  11. G. Györgyi, First-order transition to perfect generalization in a neural network with binary synapses, Phys. Rev. A 41, 7097 (1990).
  12. R. Koebarle and W. K. Theumann, Neural Networks and Spin Glasses—Proceedings of the Statphys 17 Workshop (World Scientific, Singapore, 1990).
  13. M. Advani, S. Lahiri, and S. Ganguli, Statistical mechanics of complex neural systems and high dimensional data, J. Stat. Mech. (2013) P03014.
  14. J. J. Hopfield, Neural networks and physical systems with emergent collective computational abilities, Proc. Natl. Acad. Sci. USA 79, 2554 (1982).
  15. W. A. Little, The existence of persistent states in the brain, Math. Biosci. 19, 101 (1974).
  16. D. J. Amit, H. Gutfreund, and H. Sompolinsky, Spin-glass models of neural networks, Phys. Rev. A 32, 1007 (1985).
  17. D. J. Amit, H. Gutfreund, and H. Sompolinsky, Statistical mechanics of neural networks near saturation, Ann. Phys. (NY) 173, 30 (1987).
  18. G. Montúfar, Restricted Boltzmann machines: Introduction and review, in Information Geometry and Its Applications: On the Occasion of Shun-ichi Amari's 80th Birthday, IGAIA IV Liblice, Czech Republic, June 2016 (Springer, Berlin, 2018), pp. 75–115.
  19. C. Bény, Deep learning and the renormalization group, arXiv:1301.3124.
  20. P. Mehta and D. J. Schwab, An exact mapping between the variational renormalization group and deep learning, arXiv:1410.3831.
  21. M. D. Zeiler and R. Fergus, Visualizing and understanding convolutional networks, in Computer Vision–ECCV 2014, Lecture Notes in Computer Science Vol. 8689 (Springer, Cham, 2014).
  22. H. W. Lin, M. Tegmark, and D. Rolnick, Why does deep and cheap learning work so well? J. Stat. Phys. 168, 1223 (2017).
  23. S. Iso, S. Shiba, and S. Yokoo, Scale-invariant feature extraction of neural network and renormalization group flow, Phys. Rev. E 97, 053304 (2018).
  24. E. de Mello Koch, R. de M. Koch, and L. Cheng, Is deep learning a renormalization group flow? IEEE Access 8, 106487 (2020).
  25. S. S. Funai and D. Giataganas, Thermodynamics and feature extraction by machine learning, Phys. Rev. Res. 2, 033415 (2020).
  26. F. Gong and Z. Xia, Interpreting deep learning by establishing a rigorous corresponding relationship with renormalization group, arXiv:2212.00005.
  27. D. A. Roberts, S. Yaida, and B. Hanin, The Principles of Deep Learning Theory (Cambridge University Press, Cambridge, 2022).
  28. A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, and Y. LeCun, The loss surfaces of multilayer networks, in Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research Vol. 38 (PMLR, Brookline, MA, USA, 2015), pp. 192–204.
  29. S. S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein, Deep information propagation, arXiv:1611.01232.
  30. B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli, Exponential expressivity in deep neural networks through transient chaos, in Advances in Neural Information Processing Systems (NeurIPS) (Curran Associates, Inc., Red Hook, NY, USA, 2016), Vol. 29, pp. 3360–3368.
  31. D. Oprisa and P. Toth, Criticality & deep learning II: Momentum renormalisation group, arXiv:1705.11023.
  32. D. Oprisa and P. Toth, Criticality and deep learning I: Theory vs empirics, arXiv:1702.08039.
  33. Z. Liao, R. Couillet, and M. W. Mahoney, A random matrix analysis of random Fourier features: Beyond the Gaussian kernel, a precise phase transition, and the corresponding double descent, in Advances in Neural Information Processing Systems (NeurIPS) (Curran Associates, Inc., Red Hook, NY, USA, 2020), Vol. 33, pp.–13939–13950.
  34. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, Scaling laws for neural language models, arXiv:2001.08361.
  35. E. Caballero, K. Gupta, I. Rish, and D. Krueger, Broken neural scaling laws, arXiv:2210.14891.
  36. Y. Bahri, E. Dyer, J. Kaplan, J. Lee, and U. Sharma, Explaining neural scaling laws, Proc. Natl. Acad. Sci. USA 121, e2311878121 (2024).
  37. B. Sorscher, R. Geirhos, S. Shekhar, S. Ganguli, and A. Morcos, Beyond neural scaling laws: Beating power law scaling via data pruning, in Advances in Neural Information Processing Systems (NeurIPS) (Curran Associates, Inc., Red Hook, NY, USA, 2022), Vol. 35, pp. 19523–19536.
  38. T. Hastie, A. Montanari, S. Rosset, and R. J. Tibshirani, Surprises in high-dimensional ridgeless least squares interpolation, Ann. Statist. 50, 949 (2022).
  39. L. Ziyin and M. Ueda, Exact phase transitions in deep learning, arXiv:2205.12510.
  40. B. Bordelon, A. Atanasov, and C. Pehlevan, A dynamical model of neural scaling laws, arXiv:2402.01092.
  41. Q. Ma, H. Mao, J. Liu, Z. Zhang, C. Feng, Y. Song, Y. Shao, and Y. Ma, Do neural scaling laws exist on graph self-supervised learning? arXiv:2408.11243.
  42. Z. Du, A. Zeng, Y. Dong, and J. Tang, Understanding emergent abilities of language models from the loss perspective, arXiv:2403.15796.
  43. I. Ernst, Beitrag zur Theorie des Ferromagnetismus, Z. Phys. A: Hadrons Nucl. 31, 253 (1925).
  44. S. Ruder, An overview of gradient descent optimization algorithms, arXiv:1609.04747.
  45. L. Alzubaidi, J. Zhang, A. J. Humaidi, A. Al-Dujaili, Y. Duan, O. Al-Shamma, J. Santamaría, M. A. Fadhel, M. Al-Amidie, and L. Farhan, Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions, J. Big. Data. 8, 53 (2021).
  46. W. S. McCulloch and W. Pitts, A logical calculus of the ideas immanent in nervous activity, Bull. Math. Biophys. 5, 115 (1943).
  47. A. Dongare, R. Kharde, A. D. Kachare et al., Introduction to artificial neural network, Int. J. Eng. Innov. Technol. 2, 189 (2012).
  48. S. Amato, B. Apolloni, G. Caporali, U. Madesani, and A. Zanaboni, Simulated annealing approach in backpropagation, Neurocomputing 3, 207 (1991).
  49. R. M. Burton and G. J. Mpitsos, Event-dependent control of noise enhances learning in neural networks, Neural Netw. 5, 627 (1992).
  50. M. Welling and Y. Teh, Bayesian learning via stochastic gradient Langevin dynamics, in Proceedings of the 28th International Conference on Machine Learning, ICML 2011 (ACM, New York, 2011), p. 681.
  51. T. Chen, E. Fox, and C. Guestrin, Stochastic gradient Hamiltonian Monte Carlo, in International Conference on Machine Learning, Proceedings of Machine Learning Research Vol. 32 (PMLR, Brookline, MA, USA, 2014), pp. 1683–1691.
  52. C. Li, C. Chen, D. Carlson, and L. Carin, Preconditioned stochastic gradient Langevin dynamics for deep neural networks, in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI, Washington, 2016), Vol. 30.
  53. H. Risken, Langevin equations, The Fokker-Planck Equation: Methods of Solution and Applications (Springer, Berlin, 1996), pp. 32–62.
  54. Q. Li and H. Sompolinsky, Statistical mechanics of deep linear neural networks: The backpropagating kernel renormalization, Phys. Rev. X 11, 031059 (2021).
  55. G. A. Pavliotis, The Langevin equation, Stochastic Processes and Applications: Diffusion Processes, the Fokker-Planck and Langevin Equations (Springer, New York, 2014), pp. 181–233.
  56. G. E. Hinton and R. R. Salakhutdinov, Reducing the dimensionality of data with neural networks, Science 313, 504 (2006).
  57. A. Krizhevsky, I. Sutskever, and G. E. Hinton, Imagenet classification with deep convolutional neural networks, in Advances in Neural Information Processing Systems (NeurIPS) (Curran Associates, Inc., Red Hook, NY, USA, 2012), Vol. 25, pp. 1097–1105.
  58. G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath et al., Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups, IEEE Signal Process. Mag. 29, 82 (2012).
  59. D. Ciregan, U. Meier, and J. Schmidhuber, Multi-column deep neural networks for image classification, in 2012 IEEE Conference on Computer Vision and Pattern Recognition (IEEE, Piscataway, NJ, 2012), pp. 3642–3649.
  60. S. Sun, W. Chen, L. Wang, X. Liu, and T.-Y. Liu, On the depth of deep neural networks: A theoretical view, in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI, Washington, 2016), Vol. 30.
  61. L. D. Faddeev and V. N. Popov, Feynman diagrams for the Yang-Mills field, Phys. Lett. B 25, 29 (1967).
  62. Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, Gradient-based learning applied to document recognition, Proc. IEEE 86, 2278 (1998).
  63. Z. Lu, H. Pu, F. Wang, Z. Hu, and L. Wang, The expressive power of neural networks: A view from the width, in Advances in Neural Information Processing Systems (NeurIPS) (Curran Associates, Inc., Red Hook, NY, USA, 2017), Vol. 30, pp. 6231–6239.
  64. B. Hanin and M. Sellke, Approximating continuous functions by ReLU nets of minimal width, arXiv:1710.11278.
  65. J. Johnson, Deep, skinny neural networks are not universal approximators, arXiv:1810.00393.
  66. P. Kidger and T. Lyons, Universal approximation with deep narrow networks, in Proceedings of Thirty Third Conference on Learning Theory, Proceedings of Machine Learning Research Vol. 125 (PMLR, Brookline, MA, USA, 2020), pp. 2306–2327.
  67. S. Park, C. Yun, J. Lee, and J. Shin, Minimum width for universal approximation, arXiv:2006.08859.
  68. S. Sharma, S. Sharma, and A. Athaiya, Activation functions in neural networks, Int. J. Eng. Appl. Sci. Technol. 4, 310 (2020).
  69. H. Wang, S. Ma, L. Dong, S. Huang, H. Wang, L. Ma, F. Yang, R. Wang, Y. Wu, and F. Wei, BitNet: Scaling 1-bit transformers for large language models, arXiv:2310.11453.
  70. S. Ma, H. Wang, L. Ma, L. Wang, W. Wang, S. Huang, L. Dong, R. Wang, J. Xue, and F. Wei, The era of 1-bit LLMs: All large language models are in 1.58 bits, arXiv:2402.17764.
  71. B. Hu, Introduction to real-space renormalization-group methods in critical and chaotic phenomena, Phys. Rep. 91, 233 (1982).
  72. J. Cardy, Scaling and Renormalization in Statistical Physics, Cambridge Lecture Notes in Physics Vol. 5 (Cambridge University Press, Cambridge, 1996).
  73. D. Lee, H.-S. Lee, and J. Yi (unpublished).
  74. S. F. Edwards and P. W. Anderson, Theory of spin glasses, J. Phys. F 5, 965 (1975).
  75. H. Nishimori, Statistical Physics of Spin Glasses and Information Processing: An Introduction, Mathematical and Statistical Physics Vol. 111 (Clarendon, Oxford, 2001).

Outline

Information

Phys. Rev. Research 7, 023276– Published 17 June, 2025
  • Received 6 October 2024
  • Accepted 30 May 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation