- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Data variability in neural network Bayesian inference

Javed Lindner1,2,3,*, David Dahmen1, Michael Krämer3, and Moritz Helias1,2

  • *Contact author: ja.lindner@fz-juelich.de
PDF

Phys. Rev. Research 8, 013320 – Published 25 March, 2026

DOI: https://doi.org/10.1103/r1s3-lr3g

Abstract

Bayesian inference and kernel methods are well established in machine learning. The neural network Gaussian process in particular provides a concept to investigate neural networks in the limit of infinitely wide hidden layers by using kernel and inference methods. Here, we build upon this limit and provide a field theoretic formalism that covers the generalization properties of infinitely wide networks. We systematically compute generalization properties of linear, nonlinear, and deep nonlinear networks for kernel matrices with heterogeneous entries. In contrast to currently employed spectral methods, we derive the generalization properties from the statistical properties of the input, elucidating the interplay of input dimensionality, size of the training dataset, and variability of the data. We show that data variability leads to a non-Gaussian action reminiscent of a φ3+φ4 theory. Using our formalism on a synthetic task and on MNIST, we obtain a homogeneous kernel matrix approximation for the learning curve as well as corrections due to data variability that allow the estimation of the generalization properties and exact results for the bounds of the learning curves in the case of infinitely many training data points.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (60)

  1. H. Sompolinsky, A. Crisanti, and H. J. Sommers, Chaos in random neural networks, Phys. Rev. Lett. 61, 259 (1988).
  2. D. J. Amit, H. Gutfreund, and H. Sompolinsky, Storing infinite numbers of patterns in a spin-glass model of neural networks, Phys. Rev. Lett. 55, 1530 (1985).
  3. E. Gardner, The space of interactions in neural network models, J. Phys. A: Math. Gen. 21, 257 (1988).
  4. E. Gardner and B. Derrida, Optimal storage properties of neural network models, J. Phys. A: Math. Gen. 21, 271 (1988).
  5. R. M. Neal, Bayesian Learning for Neural Networks (Springer, New York, 1996).
  6. C. Williams, Computing with infinite networks, in Advances in Neural Information Processing Systems, edited by M. Mozer, M. Jordan, and T. Petsche (MIT Press, Cambridge, MA, USA, 1996), Vol. 9.
  7. B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli, Exponential expressivity in deep neural networks through transient chaos, in Advances in Neural Information Processing Systems, Vol. 29 (Curran Associates, Inc., Barcelona, Spain, 2016).
  8. S. S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein, Deep information propagation, in 5th International Conference on Learning Representations (2017).
  9. G. Yang, Wide feedforward or recurrent neural networks of any architecture are Gaussian processes, in Advances in Neural Information Processing Systems, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett (Curran Associates, Inc., Vancouver, British Columbia, Canada, 2019), Vol. 32.
  10. H. Cui, B. Loureiro, F. Krzakala, and L. Zdeborová, Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime*, J. Stat. Mech. (2022) 114004.
  11. B. S. Ruben and C. Pehlevan, Learning Curves for Heterogeneous Feature-Subsampled Ridge Ensembles (New Orleans, Louisiana, USA, 2023).
  12. A. Jacot, F. Gabriel, and C. Hongler, Neural Tangent Kernel: Convergence and Generalization in Neural Networks (Long Beach, CA, USA, 2018).
  13. O. Cohen, O. Malka, and Z. Ringel, Learning curves for overparametrized deep neural networks: A field theory perspective, Phys. Rev. Res. 3, 023034 (2021).
  14. G. Naveh and Z. Ringel, A self consistent theory of Gaussian processes captures feature learning effects in finite CNNs, in Advances in Neural Information Processing Systems, Vol. 34, edited by A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Curran Associates, Inc., Virtual, 2021), pp. 21352–21364.
  15. J. A. Zavatone-Veth and C. Pehlevan, Exact marginal prior distributions of finite Bayesian neural networks, in Advances in Neural Information Processing Systems, Vol. 34, edited by A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Curran Associates, Inc., Virtual, 2021), pp. 3364–3375.
  16. J. A. Zavatone-Veth, A. Canatar, B. Ruben, and C. Pehlevan, Asymptotics of representation learning in finite Bayesian neural networks, in Advances in Neural Information Processing Systems, Vol. 34, edited by A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (NeurIPS, virtual, 2021).
  17. Q. Li and H. Sompolinsky, Statistical mechanics of deep linear neural networks: The backpropagating kernel renormalization, Phys. Rev. X 11, 031059 (2021).
  18. D. A. Roberts, S. Yaida, and B. Hanin, The Principles of Deep Learning Theory (Cambridge University Press, 2022).
  19. B. Bordelon and C. Pehlevan, Self-consistent dynamical field theory of kernel evolution in wide neural networks, in Advances in Neural Information Processing Systems, edited by S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Curran Associates, Inc., New Orleans, Louisiana, USA, 2022), pp. 32240–32256.
  20. K. P. Murphy, Probabilistic Machine Learning: Advanced Topics (MIT Press, 2023).
  21. Y. LeCun, C. Cortes, and C. J. Burges, The MNIST database of handwritten digits (1998).
  22. D. J. MacKay, Information Theory, Inference and Learning Algorithms (Cambridge University Press, 2003).
  23. K. P. Murphy, Probabilistic Machine Learning: An Introduction (MIT Press, 2022).
  24. C. K. I. Williams and D. Barber, Bayesian classification with Gaussian processes, IEEE Trans. Pattern Anal. Mach. Intel. 20, 1342 (1998).
  25. C. Rasmussen and C. Williams, Gaussian Processes for Machine Learning, Adaptive Computation and Machine Learning (MIT Press, Cambridge, MA, 2006), p. 248.
  26. Y. Cho and L. Saul, Kernel methods for deep learning, in Advances in Neural Information Processing Systems, edited by Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta (Curran Associates, Inc., 2009), Vol. 22.
  27. J. Lee, J. Sohl-Dickstein, J. Pennington, R. Novak, S. Schoenholz, and Y. Bahri, Deep neural networks as Gaussian processes, in International Conference on Learning Representations (OpenReview.net, Vancouver, British Columbia, Canada, 2018).
  28. K. Segadlo, B. Epping, A. van Meegen, D. Dahmen, M. Krämer, and M. Helias, Unified field theoretical approach to deep and recurrent neuronal networks, J. Stat. Mech. (2022) 103401.
  29. S. Ariosto, R. Pacelli, M. Pastore, F. Ginelli, M. Gherardi, and P. Rotondo, Statistical mechanics of deep learning beyond the infinite-width limit, Nat. Mach. Intell. 5, 1497 (2023).
  30. See Supplemental Material at http://link.aps.org/supplemental/10.1103/r1s3-lr3g for detailed versions of the calculations performed in the main text and Appendix.
  31. A. E. Hoerl and R. W. Kennard, Ridge regression: Biased estimation for nonorthogonal problems, Technometrics 42, 80 (2000).
  32. C. K. Williams and C. E. Rasmussen, Gaussian Processes for Machine Learning, 1st ed. (MIT Press, Cambridge, 2006).
  33. I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning (MIT Press, 2016).
  34. C. De Dominicis, Dynamics as a substitute for replicas in systems with quenched random impurities, Phys. Rev. B 18, 4913 (1978).
  35. P. Martin, E. Siggia, and H. Rose, Statistical dynamics of classical systems, Phys. Rev. A 8, 423 (1973).
  36. H.-K. Janssen, On a Lagrangean for classical field dynamics and renormalization group calculations of dynamical critical properties, Z. Phys. B 23, 377 (1976).
  37. J. Stapmanns, T. Kühn, D. Dahmen, T. Luu, C. Honerkamp, and M. Helias, Self-consistent formulations for stochastic nonlinear neuronal dynamics, Phys. Rev. E 101, 042124 (2020).
  38. M. Helias and D. Dahmen, Statistical Field Theory for Neural Networks (Springer International, 2020), p. 203.
  39. K. Fischer and J. Hertz, Spin Glasses (Cambridge University Press, 1991).
  40. M. Mezard and A. Montanari, Information, Physics and Computation (Oxford University Press, Oxford, UK, 2009).
  41. J. Zinn-Justin, Quantum Field Theory and Critical Phenomena (Clarendon Press, Oxford, 1996).
  42. T. Ensslin and M. Frommert, Reconstruction of signals with unknown spectra in information field theory with parameter uncertainty, Phys. Rev. D 83, 105014 (2011).
  43. E. Dyer and G. Gur-Ari, Asymptotics of wide networks from Feynman diagrams, arXiv:1909.11304
  44. K. Fischer, A. René, C. Keup, M. Layer, D. Dahmen, and M. Helias, Decomposing neural networks as mappings of correlation functions, Phys. Rev. Res. 4, 043143 (2022).
  45. K. Segadlo, Theory of learning and prediction by deep and recurrent networks in gaussian process approximation, Master's thesis, RWTH Aachen University, 2021.
  46. J. Lee, Y. Bahri, R. Novak, S. S. Schoenholz, J. Pennington, and J. Sohl-Dickstein, Deep neural networks as Gaussian processes, arXiv:1711.00165.
  47. C. K. Williams, Computation with infinite neural networks, Neural Comput. 10, 1203 (1998).
  48. R. Price, A useful theorem for nonlinear devices having Gaussian inputs, IRE Trans. Inf. Theory 4, 69 (1958).
  49. J. Schuecker, S. Goedeke, D. Dahmen, and M. Helias, Functional methods for disordered neural networks, arXiv:1605.06758.
  50. L. Molgedey, J. Schuchhardt, and H. Schuster, Suppressing chaos in neural networks by noise, Phys. Rev. Lett. 69, 3717 (1992).
  51. L. Xiao, Y. Bahri, J. Sohl-Dickstein, S. S. Schoenholz, and J. Pennington, Dynamical isometry and a mean field theory of CNNs: How to train 10,000-layer vanilla convolutional neural networks, arXiv:1806.05393.
  52. I. Seroussi, G. Naveh, and Z. Ringel, Separation of scales and a thermodynamic description of feature learning in some CNNs, Nat. Commun. 14, 908 (2023).
  53. K. Fischer, J. Lindner, D. Dahmen, Z. Ringel, M. Krämer, and M. Helias, Critical feature learning in deep neural networks, arXiv:2405.10761.
  54. D. Malzahn and M. Opper, A variational approach to learning curves, in Advances in Neural Information Processing Systems, edited by T. Dietterich, S. Becker, and Z. Ghahramani (MIT Press, 2001), Vol. 14.
  55. G. Naveh, O. Ben-David, H. Sompolinsky, and Z. Ringel, Predicting the outputs of finite networks trained with noisy gradients, Phys. Rev. E 104, 064301 (2021).
  56. S. Yaida, Non-Gaussian processes and neural networks at finite widths, in Proceedings of The First Mathematical and Scientific Machine Learning Conference, Proceedings of Machine Learning Research, edited by J. Lu and R. Ward (PMLR, 2020), Vol. 107, pp. 165–192.
  57. K. Fischer, D. Dahmen, and M. Helias, Field theory for optimal signal propagation in ResNets, arXiv:2305.07715.
  58. Q. Li, B. Sorscher, and H. Sompolinsky, Representations and generalization in artificial and brain neural networks, Proc. Nat. Acad. Sci. USA 121, e2311805121 (2024).
  59. K. Fukunaga, Introduction to Statistical Pattern Recognition, 2nd ed. (Academic Press, 2009).
  60. J. Lindner, D. Dahmen, M. Krämer, and M. Helias, Data variability in neural network Bayesian inference, Zenodo, 2025, https://zenodo.org/records/17529999.

Outline

Information

Phys. Rev. Research 8, 013320– Published 25 March, 2026
  • Received 7 December 2023
  • Accepted 5 August 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation