- Physical Review Journals
All JournalsPhysics Magazine

Physical Review E

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Field theory for optimal signal propagation in residual networks

Kirsten Fischer1,2,*, David Dahmen1, and Moritz Helias1,3

  • *Contact author: ki.fischer@fz-juelich.de
PDF

Phys. Rev. E 112, 065301 – Published 1 December, 2025

DOI: https://doi.org/10.1103/5lgz-4t7h

Abstract

Residual networks have significantly better trainability and thus performance than feed-forward networks at large depth. Introducing skip connections facilitates signal propagation to deeper layers. In addition, previous works found that adding a scaling parameter for the residual branch further improves generalization performance. While they empirically identified a particularly beneficial range of values for this scaling parameter, the mechanism for the resulting performance improvement and its universality across network hyperparameters remain an open question. For feed-forward networks, finite-size theories have led to important insights with regard to signal propagation and hyperparameter tuning. We here derive a systematic finite-size field theory for residual networks to study signal propagation and its dependence on the scaling for the residual branch. We derive analytical expressions for the response function, a measure for the network's sensitivity to inputs, and show that for deep networks the empirically found values for the scaling parameter lie within the range of maximal sensitivity. Furthermore, we obtain an analytical expression for the optimal scaling parameter that depends only weakly on other network hyperparameters, such as the weight variance, thereby explaining its universality across hyperparameters. Overall, this work provides a theoretical framework to study ResNets at finite size.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (60)

  1. A. Krizhevsky, I. Sutskever, and G. E. Hinton, ImageNet classification with deep convolutional neural networks, in Advances in Neural Information Processing Systems, edited by F. Pereira, C. J. Burges, L. Bottou, and K. Q. Weinberger (Curran Associates, Red Hook, NY, 2012), Vol. 25, pp. 1097–1105.
  2. D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot et al., Mastering the game of go with deep neural networks and tree search, Nature (London) 529, 484 (2016).
  3. K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning for image recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA (IEEE, Piscataway, NJ, 2016), pp. 770–778.
  4. K. He, X. Zhang, S. Ren, and J. Sun, Identity mappings in deep residual networks, in European Conference on Computer Vision, edited by B. Leibe, J. Matas, N. Sebe, and M. Welling, Lecture Notes in Computer Science Vol. 9908 (Springer, Cham, 2016), pp. 630–645.
  5. A. Krizhevsky and G. Hinton, Learning multiple layers of features from tiny images, Technical Report, Department of Computer Science, University of Toronto, 2009.
  6. C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, Inception-v4, inception-resnet and the impact of residual connections on learning, in Proceedings of the Thirty-first AAAI Conference on Artificial Intelligence, AAAI'17 (AAAI Press, Reston, VA, 2017), pp. 4278–4284.
  7. S. Ioffe and C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, in Proceedings of the 32nd International Conference on Machine Learning, edited by F. Bach and D. Blei, Proceedings of Machine Learning Research Vol. 37 (JMLR, Lille, France, 2015), pp. 448–456.
  8. K. Huang, Y. Wang, M. Tao, and T. Zhao, Why do deep residual networks generalize better than deep feedforward networks? A neural tangent kernel perspective, in Advances in Neural Information Processing Systems, edited by H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Curran Associates, Red Hook, NY, 2020), Vol. 33, pp. 2698–2709.
  9. A. Jacot, F. Gabriel, and C. Hongler, Neural tangent kernel: Convergence and generalization in neural networks, in Advances in Neural Information Processing Systems (Curran Associates, Red Hook, NY, 2018), Vol. 31, pp. 8580–8589.
  10. T. Bachlechner, B. P. Majumder, H. Mao, G. Cottrell, and J. McAuley, Rezero is all you need: Fast convergence at large depth, in Proceedings of the Thirty-seventh Conference on Uncertainty in Artificial Intelligence, edited by C. de Campos and M. H. Maathuis, Proceedings of Machine Learning Research Vol. 161 (JMLR, Cambridge, MA, 2021), pp. 1352–1361.
  11. T. Tirer, J. Bruna, and R. Giryes, Kernel-based smoothness analysis of residual networks, in Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, edited by J. Bruna, J. Hesthaven, and L. Zdeborova, Proceedings of Machine Learning Research Vol. 145 (JMLR, Cambridge, MA, 2022), pp. 921–954.
  12. D. Barzilai, A. Geifman, M. Galun, and R. Basri, A kernel perspective of skip connections in convolutional networks, in The Eleventh International Conference on Learning Representations (ICLR, 2023).
  13. J. Zhang, B. Han, L. Wynter, B. K. H. Low, and M. Kankanhalli, Towards robust ResNet: A small step but a giant leap, in Proceedings of the Twenty-eighth International Joint Conference on Artificial Intelligence, IJCAI-19 (International Joint Conferences on Artificial Intelligence Organization, 2019), pp. 4285–4291.
  14. D. Arpit, V. Campos, and Y. Bengio, How to initialize your network? Robust initialization for weightnorm & ResNets, in Advances in Neural Information Processing Systems, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Curran Associates, Red Hook, NY, 2019), Vol. 32.
  15. S. Hayou, E. Clerico, B. He, G. Deligiannidis, A. Doucet, and J. Rousseau, Stable ResNet, in Proceedings of the 24th International Conference on Artificial Intelligence and Statistics, edited by A. Banerjee and K. Fukumizu, Proceedings of Machine Learning Research Vol. 130 (JMLR, Cambridge, MA, Cambridge, MA, 2021), pp. 1324–1332.
  16. S. Hayou, J.-F. Ton, A. Doucet, and Y. W. Teh, Robust pruning at initialization, in International Conference on Learning Representations (ICLR, 2021).
  17. H. Zhang, D. Yu, M. Yi, W. Chen, and T.-Y. Liu, Stabilize deep ResNet with a sharp scaling factor τ, Mach. Learn. 111, 3359 (2022).
  18. B. Bordelon, L. Noci, M. B. Li, B. Hanin, and C. Pehlevan, Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit, in The Twelfth International Conference on Learning Representations (ICLR, 2024).
  19. G. Yang, E. J. Hu, I. Babuschkin, S. Sidor, X. Liu, D. Farhi, N. Ryder, J. Pachocki, W. Chen, and J. Gao, Tuning large neural networks via zero-shot hyperparameter transfer, in Advances in Neural Information Processing Systems, edited by A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Curran Associates, Red Hook, NY, 2021).
  20. J. Halverson, A. Maiti, and K. Stoner, Neural networks and quantum field theory, Mach. Learn.: Sci. Technol. 2, 035002 (2021).
  21. G. Naveh, O. Ben David, H. Sompolinsky, and Z. Ringel, Predicting the outputs of finite deep neural networks trained with noisy gradients, Phys. Rev. E 104, 064301 (2021).
  22. K. Segadlo, B. Epping, A. van Meegen, D. Dahmen, M. Krämer, and M. Helias, Unified field theoretical approach to deep and recurrent neuronal networks, J. Stat. Mech. (2022) 103401.
  23. D. A. Roberts, S. Yaida, and B. Hanin, The Principles of Deep Learning Theory (Cambridge University Press, Cambridge, 2022).
  24. G. Naveh and Z. Ringel, A self consistent theory of gaussian processes captures feature learning effects in finite CNNS, in Advances in Neural Information Processing Systems, edited by A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Curran Associates, Red Hook, NY, 2021).
  25. J. A. Zavatone-Veth and C. Pehlevan, Exact marginal prior distributions of finite Bayesian neural networks, in Advances in Neural Information Processing Systems, edited by A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Curran Associates, Red Hook, NY, 2021).
  26. J. A. Zavatone-Veth, W. L. Tong, and C. Pehlevan, Contrasting random and learned features in deep Bayesian linear regression, Phys. Rev. E 105, 064118 (2022).
  27. I. Seroussi, G. Naveh, and Z. Ringel, Separation of scales and a thermodynamic description of feature learning in some CNNs, Nat. Commun. 14, 908 (2023).
  28. A. X. Yang, M. Robeyns, E. Milsom, B. Anson, N. Schoots, and L. Aitchison, A theory of representation learning gives a deep generalisation of kernel methods, in Proceedings of the 40th International Conference on Machine Learning, edited by A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, Proceedings of Machine Learning Research Vol. 202 (JMLR, Cambridge, MA, 2023), pp. 39380–39415.
  29. K. Fischer, J. Lindner, D. Dahmen, Z. Ringel, M. Krämer, and M. Helias, Critical feature learning in deep neural networks, in Proceedings of the 41st International Conference on Machine Learning, edited by R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, Proceedings of Machine Learning Research Vol. 235 (JMLR, Cambridge, MA, 2024), pp. 13660–13690.
  30. N. Rubin, I. Seroussi, and Z. Ringel, Grokking as a first order phase transition in two layer networks, in The Twelfth International Conference on Learning Representations (ICLR, 2024).
  31. J. Lindner, D. Dahmen, M. Krämer, and M. Helias, A theory of data variability in neural network Bayesian inference, arXiv:2307.16695.
  32. B. Bordelon and C. Pehlevan, Self-consistent dynamical field theory of kernel evolution in wide neural networks, J. Stat. Mech. (2023) 114009.
  33. K. Fischer, D. Dahmen, and M. Helias, Field theory for optimal signal propagation in ResNets, Zenodo (2023), 10.5281/zenodo.17395886.
  34. J. Zinn-Justin, Quantum Field Theory and Critical Phenomena (Clarendon Press, Oxford, 1996).
  35. J. Hertz, A. Krogh, and R. G. Palmer, Introduction to the Theory of Neural Computation (Westview Press Inc, 1991).
  36. S. S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein, Deep information propagation, International Conference on Learning Representations (OpenReview.net, 2017).
  37. A. Papoulis and S. U. Pillai, Probability, Random Variables, and Stochastic Processes, 4th ed. (McGraw-Hill, Boston, 2002).
  38. G. Yang and S. Schoenholz, Mean field residual networks: On the edge of chaos, in Advances in Neural Information Processing Systems, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Curran Associates, Red Hook, NY, 2017), Vol. 30.
  39. A. Bukva, J. de Gier, K. T. Grosvenor, R. Jefferson, K. Schalm, and E. Schwander, Criticality versus uniformity in deep neural networks, arXiv:2304.04784.
  40. L. Chizat, E. Oyallon, and F. Bach, On lazy training in differentiable programming, in Advances in Neural Information Processing Systems (Curran Associates, Red Hook, NY, 2019), Vol. 32.
  41. M. Welling and Y. W. Teh, Bayesian learning via stochastic gradient langevin dynamics, in Proceedings of the 28th International Conference on International Conference on Machine Learning (ICLR, 2011), pp. 681–688.
  42. S. Mandit, M. D. Hoffman, and D. M. Blei, Stochastic gradient descent as approximate Bayesian inference, J. Mach. Learn. Res. 18, 1 (2017).
  43. S. Li, J. Jiao, Y. Han, and T. Weissman, Demystifying ResNet, arXiv:1611.01186.
  44. Y. Furusho and K. Ikeda, Resnet and batch-normalization improve data separability, in Proceedings of the Eleventh Asian Conference on Machine Learning, edited by W. S. Lee and T. Suzuki, Proceedings of Machine Learning Research Vol. 101 (JMLR, 2019), pp. 94–108.
  45. B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli, Exponential expressivity in deep neural networks through transient chaos, in Advances in Neural Information Processing Systems 29, edited by D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (Curran Associates, Red Hook, NY, 2016), pp. 3360–3368.
  46. B. Hanin, Which neural net architectures give rise to exploding and vanishing gradients?, in Advances in Neural Information Processing Systems, edited by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Curran Associates, Red Hook, NY, 2018), Vol. 31.
  47. R. T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud, Neural ordinary differential equations, in Advances in Neural Information Processing Systems 31, edited by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Curran Associates, Red Hook, NY, 2018), pp. 6571–6583.
  48. A.-S. Cohen, R. Cont, A. Rossier, and R. Xu, Scaling properties of deep residual networks, in Proceedings of the 38th International Conference on Machine Learning, edited by M. Meila and T. Zhang, Proceedings of Machine Learning Research Vol. 139 (ICLR, 2021), pp. 2039–2048.
  49. R. Barboni, G. Peyré, and F.-X. Vialard, On global convergence of resnets: From finite to infinite width using linear parameterization, in Advances in Neural Information Processing Systems, edited by S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Curran Associates, Red Hook, NY, 2022), Vol. 35, pp. 16385–16397.
  50. R. Barboni, G. Peyré, and F.-X. Vialard, Understanding the training of infinitely deep and wide ResNets with conditional optimal transport, Comm. Pure Appl. Math. 78, 2149 (2025).
  51. Y. Chen, F. Liu, Y. Lu, G. Chrysos, and V. Cevher, Generalization of scaled deep ResNets in the mean-field regime, in The Twelfth International Conference on Learning Representations (ICLR, 2024).
  52. M. B. Li and M. Nica, Differential equation scaling limits of shaped and unshaped neural networks, Transactions on Machine Learning Research (OpenReview.net, 2024).
  53. P. Marion, A. Fermanian, G. Biau, and J.-P. Vert, Scaling ResNets in the large-depth regime, J. Mach. Learn. Res. 26, 1 (2025).
  54. D. Balduzzi, M. Frean, L. Leary, J. P. Lewis, K. W.-D. Ma, and B. McWilliams, The shattered gradients problem: If ResNets are the answer, then what is the question?, in Proceedings of the 34th International Conference on Machine Learning, edited by D. Precup and Y. W. Teh, Proceedings of Machine Learning Research Vol. 70 (JMLR, Cambridge, MA, 2017), pp. 342–350.
  55. A. Zaeemzadeh, N. Rahnavard, and M. Shah, Norm-preservation: Why residual networks can become extremely deep? IEEE Trans. Pattern Anal. Mach. Intell. 43, 3980 (2021).
  56. Z. Ling and R. C. Qiu, Spectrum concentration in deep residual learning: A free probability approach, IEEE Access, 7, 105212 (2019).
  57. X. Zhang, R. Jiang, W. Gao, R. Willett, and M. Maire, Residual connections harm abstract feature learning in masked autoencoders, arXiv:2404.10947.
  58. D. Doshi, T. He, and A. Gromov, Critical initialization of wide and deep neural networks using partial Jacobians: General theory and applications, in Advances in Neural Information Processing Systems 36, edited by A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Curran Associates, Red Hook, NY, 2023), pp. 40054–40095.
  59. R. Price, A useful theorem for nonlinear devices having Gaussian inputs, IRE Trans. Inf. Theory 4, 69 (1958).
  60. N. Rubin, K. Fischer, J. Lindner, I. Seroussi, Z. Ringel, M. Krämer, and M. Helias, From kernels to features: A multi-scale adaptive theory of feature learning, in Proceedings of the 42nd International Conference on Machine Learning, edited by A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu, Proceedings of Machine Learning Research Vol. 267 (PMLR, 2025), pp. 52225–52257.

Outline

Information

Phys. Rev. E 112, 065301– Published 1 December, 2025
  • Received 27 August 2024
  • Accepted 17 October 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation