- Physical Review Journals
All JournalsPhysics Magazine

Physical Review E

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Featured in Physics
  • Open Access

Statistical mechanics of support vector regression

Abdulkadir Canatar and SueYeon Chung

PDF

Phys. Rev. E 112, 025301 – Published 4 August, 2025

DOI: https://doi.org/10.1103/78dr-c4xd

Abstract

A key problem in deep learning and computational neuroscience is relating the geometrical properties of neural representations to task performance. Here, we consider this problem for continuous decoding tasks where neural variability may affect task precision. Using methods from statistical mechanics, we study the average-case learning curves for ɛ-insensitive support vector regression and discuss its capacity as a measure of linear decodability. Our analysis reveals a phase transition in training error at a critical load, capturing the interplay between the tolerance parameter ɛ and neural variability. We uncover a double-descent phenomenon in the generalization error, showing that ɛ acts as a regularizer, both suppressing and shifting these peaks. Theoretical predictions are validated both with toy models and deep neural networks, extending the theory of support vector machines to continuous tasks with inherent neural variability.

View figure in article

Physics Subject Headings (PhySH)

Collections

This article appears in the following collection:

Statistical Physics Meets Machine Learning - Machine Learning Meets Statistical Physics

The Editors of Physical Review E are pleased to present the Collection on Statistical Physics Meets Machine Learning - Machine Learning Meets Statistical Physics, highlighting research at the intersection of machine learning and statistical physics, on the occasion of the two Statistical Physics Meets Machine Learning and the two Machine Learning Meets Statistical Physics sessions at the 2025 Global Physics Summit. The Collection is being guest edited by David Schwab (CUNY, New York) and Yuhai Tu (IBM Watson Research Center Yorktown Heights, NY). Every article published in this collection underwent a rigorous peer review process, adhering to the same high standards applied to all papers. The Physical Review E editorial team managed the peer review and made all editorial decisions.

Viewpoint

Viewing Neural Networks Through a Statistical-Physics Lens

Published 23 February, 2026

Statistical physics is shedding light on how network architecture and data structure shape the effectiveness of neural-network learning.

See more in Physics

Article Text

References (48)

  1. T. A. Poggio and F. Anselmi, Visual Cortex and Deep Networks: Learning Invariant Representations (MIT Press, Cambridge, MA, 2016)
  2. Y. LeCun, Y. Bengio, and G. Hinton, Nature (London) 521, 436 (2015).
  3. I. Goodfellow, H. Lee, Q. Le, A. Saxe, and A. Ng, Adv. Neural Inf. Process. Syst. 22, 646 (2009).
  4. H. Hong, D. L. Yamins, N. J. Majaj, and J. J. DiCarlo, Nat. Neurosci. 19, 613 (2016).
  5. H. Drucker, C. J. C. Burges, L. Kaufman, A. Smola, and V. Vapnik, in Advances in Neural Information Processing Systems, edited by M. Mozer, M. Jordan, and T. Petsche (MIT Press, Cambridge, MA, 1996), Vol. 9.
  6. B. Schölkopf, A. J. Smola, R. C. Williamson, and P. L. Bartlett, Neural Comput. 12, 1207 (2000).
  7. M. Belkin, D. Hsu, S. Ma, and S. Mandal, Proc. Natl. Acad. Sci. USA 116, 15849 (2019).
  8. M. Mézard, G. Parisi, and M. A. Virasoro, Spin Glass Theory and Beyond: An Introduction to the Replica Method and Its Applications (World Scientific Publishing Company, Singapore, 1987), Vol. 9.
  9. A. Engel, Statistical Mechanics of Learning (Cambridge University Press, Cambridge, UK, 2001).
  10. M. Mezard and A. Montanari, Information, Physics, and Computation (Oxford University Press, Oxford, UK, 2009).
  11. B. Bordelon, A. Canatar, and C. Pehlevan, in Proceedings of the International Conference on Machine Learning (PMLR, Cambridge, MA, 2020), pp. 1024–1034.
  12. A. Jacot, B. Simsek, F. Spadaro, C. Hongler, and F. Gabriel, in Proceedings of the International Conference on Machine Learning (PMLR, Cambridge, MA, 2020), pp. 4631–4640.
  13. A. Canatar, B. Bordelon, and C. Pehlevan, Nat. Commun. 12, 2914 (2021).
  14. J. B. Simon, M. Dickens, D. Karkada, and M. Deweese, Trans. Mach. Learn. Res. (2023).
  15. A. Atanasov, B. Bordelon, S. Sainathan, and C. Pehlevan, in Proceedings of the 11th International Conference on Learning Representations (2022).
  16. A. B. Atanasov, J. A. Zavatone-Veth, and C. Pehlevan, arXiv:2405.00592.
  17. B. Loureiro, C. Gerbelot, H. Cui, S. Goldt, F. Krzakala, M. Mezard, and L. Zdeborová, Adv. Neural Inf. Process. Syst. 34, 18137 (2021).
  18. Y. Bengio, A. Courville, and P. Vincent, IEEE Trans. Pattern Anal. Mach. Intell. 35, 1798 (2013).
  19. A. Achille and S. Soatto, J. Mach. Learn. Res. 19, 1 (2018).
  20. S. Chung and L. Abbott, Curr. Opin. Neurobiol. 70, 137 (2021).
  21. S. Mei, T. Misiakiewicz, and A. Montanari, in Proceedings of the Conference on Learning Theory (PMLR, Cambridge, MA, 2021), pp. 3351–3418.
  22. A. Favero, F. Cagnetta, and M. Wyart, Adv. Neural Inf. Process. Syst. 34, 9456 (2021).
  23. B. Elesedy, Adv. Neural Inf. Process. Syst. 34, 17273 (2021).
  24. S. Y. Chung, D. D. Lee, and H. Sompolinsky, Phys. Rev. X 8, 031003 (2018).
  25. A. J. Wakhloo, T. J. Sussman, and S. Y. Chung, Phys. Rev. Lett. 131, 027301 (2023).
  26. M. Farrell, B. Bordelon, S. Trivedi, and C. Pehlevan, arXiv:2110.07472.
  27. B. Ruben and C. Pehlevan, Adv. Neural Inf. Process. Syst. 36, 50041 (2023).
  28. B. Adlam and J. Pennington, in Proceedings of the International Conference on Machine Learning (PMLR, Cambridge, MA, 2020), pp. 74–84.
  29. C. Cheng and A. Montanari, Dimension free ridge regression, Ann. Stat. 52, 2879 (2024) .
  30. S. Bös, W. Kinzel, and M. Opper, Phys. Rev. E 47, 1384 (1993).
  31. H. S. Seung and H. Sompolinsky, Proc. Natl. Acad. Sci. USA 90, 10749 (1993).
  32. N. Brunel and J.-P. Nadal, Neural Comput. 10, 1731 (1998).
  33. M. Belkin, S. Ma, and S. Mandal, in Proceedings of the International Conference on Machine Learning (PMLR, Cambridge, MA, 2018), pp. 541–549.
  34. B. Schölkopf, A. J. Smola, F. Bach et al., Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond (MIT Press, Cambridge, MA, 2002).
  35. A. Maloney, D. A. Roberts, and J. Sully, arXiv:2210.16859.
  36. D. Wu and J. Xu, Adv. Neural Inf. Process. Syst. 33, 10112 (2020).
  37. T. Hastie, The Elements of Statistical Learning: Data Mining, Inference, and Prediction (Springer, Berlin, 2009).
  38. A. Canatar, J. Feather, A. Wakhloo, and S. Chung, Adv. Neural Inf. Process. Syst. 36, 47052 (2023).
  39. C. Thrampoulidis, S. Oymak, and B. Hassibi, in Proceedings of the Conference on Learning Theory (PMLR, Cambridge, MA, 2015), pp. 1683–1709.
  40. E. Dobriban and S. Wager, Ann. Stat. 46, 247 (2018).
  41. D. Kobak, J. Lomond, and B. Sanchez, J. Mach. Learn. Res. 21, 1 (2020).
  42. T. Hastie, A. Montanari, S. Rosset, and R. J. Tibshirani, Ann. Stat. 50, 949 (2022).
  43. Z. Zhang, Machine Learning: Science and Technology 6, 025010 (2025).
  44. https://github.com/canatara/svr_code.
  45. S. Mei, T. Misiakiewicz, and A. Montanari, Appl. Comput. Harmon. Anal. 59, 3 (2022).
  46. S. Mei and A. Montanari, Commun. Pure Appl. Math. 75, 667 (2022).
  47. H. Hu and Y. M. Lu, arXiv:2205.06798.
  48. T. Misiakiewicz and B. Saeed, arXiv:2403.08938.

Outline

Information

Phys. Rev. E 112, 025301– Published 4 August, 2025
  • Received 9 January 2025
  • Accepted 23 June 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation