- Physical Review Journals
All JournalsPhysics Magazine

Physical Review E

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Landscape complexity for the empirical risk of generalized linear models: Discrimination between structured data

Theodoros G. Tsironis and Aris L. Moustakas

PDF

Phys. Rev. E 112, 065307 – Published 8 December, 2025

DOI: https://doi.org/10.1103/3mbj-xkgk

Abstract

We use the Kac-Rice formula and results from random matrix theory to obtain the average number of critical points of a family of high-dimensional empirical loss functions, where the data are correlated d-dimensional Gaussian vectors, whose number has a fixed ratio with their dimension. The correlations are introduced to model the existence of structure in the data, as is common in current machine learning systems. Under a technical hypothesis, our results are exact in the large-d limit, and characterize the annealed landscape complexity, namely the logarithm of the expected number of critical points at a given value of the loss. We first address in detail the landscape of the loss function of a single perceptron and then generalize it to the case where two competing datasets with different covariance matrices are present, with the perceptron seeking to discriminate between them. The latter model can be applied to understand the interplay between adversity and nontrivial data structures. For completeness, we also treat the case of a loss function used in training generalized linear models in the presence of correlated input data.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (68)

  1. Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, Proc. IEEE 86, 2278 (1998).
  2. H. Xiao, K. Rasul, and R. Vollgraf, arXiv:1708.07747.
  3. H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein, in Advances in Neural Information Processing Systems, Vol. 31 (Curran Associates, Inc., Red Hook, NY, 2018).
  4. A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, and Y. LeCun, in Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics (PMLR, 2015), pp. 192–204.
  5. N. P. Baskerville, J. P. Keating, F. Mezzadri, and J. Najnudel, J. Stat. Phys. 186, 29 (2022).
  6. A. Maillard, G. B. Arous, and G. Biroli, in Mathematical and Scientific Machine Learning (PMLR, 2020), pp. 287–327.
  7. Y. Arjevani and M. Field, in Advances in Neural Information Processing Systems, Vol. 33 (Curran Associates, Inc., Red Hook, NY, 2020), pp. 5441–5452.
  8. Y. Arjevani and M. Field, in Advances in Neural Information Processing Systems, Vol. 34 (Curran Associates, Inc., Red Hook, NY, 2021), pp. 15162–15174.
  9. Y. V. Fyodorov and R. Tublin, J. Phys. A: Math. Theor. 55, 244008 (2022).
  10. W. S. McCulloch and W. Pitts, Bull. Math. Biophys. 5, 115 (1943).
  11. B. Derrida, E. Gardner, and A. Zippelius, Europhys. Lett. 4, 167 (1987).
  12. E. Gardner and B. Derrida, J. Phys. A: Math. Gen. 21, 271 (1988).
  13. E. Gardner and B. Derrida, J. Phys. A: Math. Gen. 22, 1983 (1989).
  14. S. Franz and G. Parisi, J. Phys. A: Math. Theor. 49, 145001 (2016).
  15. J.-M. Azaïs and M. Wschebor, Level Sets and Extrema of Random Processes and Fields (John Wiley & Sons, New York, NY, 2009).
  16. A. Auffinger, Random matrices, complexity of spin glasses and heavy tailed processes, Ph.D. thesis, New York University, United States – New York, 2011.
  17. V. Ros and Y. V. Fyodorov, Spin Glass Theory and Far Beyond: Replica Symmetry Breaking after 40 Years 24 (2022).
  18. K. Asgari, A. Montanari, and B. Saeed, arXiv:2502.01953.
  19. D. P. Kingma and M. Welling, arXiv:1312.6114.
  20. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, in Advances in Neural Information Processing Systems, Vol. 27 (Curran Associates, Inc., Red Hook, NY, 2014).
  21. M. E. A. Seddik, C. Louart, M. Tamaazousti, and R. Couillet, in International Conference on Machine Learning (PMLR, 2020), pp. 8573–8582.
  22. S. Goldt, B. Loureiro, G. Reeves, F. Krzakala, M. Mézard, and L. Zdeborová, in Mathematical and Scientific Machine Learning (PMLR, 2022), pp. 426–471.
  23. L. Pesce, F. Krzakala, B. Loureiro, and L. Stephan, in International Conference on Machine Learning (PMLR, 2023), pp. 27680–27708.
  24. J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, in International Conference on Machine Learning (PMLR, 2015), pp. 2256–2265.
  25. Y. Song and S. Ermon, in Advances in Neural Information Processing Systems, Vol. 32 (Curran Associates, Inc., Red Hook, NY, 2019).
  26. J. Ho, A. Jain, and P. Abbeel, in Advances in Neural Information Processing Systems, Vol. 33 (Curran Associates, Inc., Red Hook, NY, 2020), p. 6840.
  27. A. Cauchy, in Comptes rendus de l'Académie des Sciences, Vol. 25 (French Academy of Sciences, 1847), p. 536.
  28. H. Robbins and D. Siegmund, in Optimizing Methods in Statistics (Elsevier, Amsterdam, 1971), pp. 233–257.
  29. M. Benaïm, in Seminaire de Probabilites XXXIII (Springer, 2006), pp. 1–68.
  30. I. Panageas and G. Piliouras, arXiv:1605.00405.
  31. I. Panageas, G. Piliouras, and X. Wang, in Advances in Neural Information Processing Systems, Vol. 32 (Curran Associates, Inc., Red Hook, NY, 2019).
  32. K. Antonakopoulos, P. Mertikopoulos, G. Piliouras, and X. Wang, in International Conference on Machine Learning (PMLR, 2022), pp. 731–771.
  33. D. Saad and S. A. Solla, Phys. Rev. Lett. 74, 4337 (1995).
  34. R. Veiga, L. Stephan, B. Loureiro, F. Krzakala, and L. Zdeborová, in Advances in Neural Information Processing Systems, Vol. 35 (Curran Associates, Inc., Red Hook, NY, 2022), p. 23244.
  35. S. S. Mannelli, F. Krzakala, P. Urbani, and L. Zdeborova, in International Conference on Machine Learning (PMLR, 2019), pp. 4333–4342.
  36. F. Mignacco, F. Krzakala, P. Urbani, and L. Zdeborová, J. Stat. Mech. (2021) 124008.
  37. Y. V. Fyodorov, Phys. Rev. Lett. 92, 240601 (2004).
  38. Y. V. Fyodorov, Acta Phys. Polonica B 36, 2699 (2005).
  39. Y. V. Fyodorov and I. Williams, J. Stat. Phys. 129, 1081 (2007).
  40. Y. V. Fyodorov and J. P. Keating, Philos. Trans. R. Soc. A 372, 20120503 (2014).
  41. G. Ben Arous, P. Bourgade, and B. McKenna, Probab. Math. Phys. 3, 731 (2023).
  42. A. Auffinger and W.-K. Chen, J. Stat. Phys. 157, 40 (2014).
  43. Z. Fan, S. Mei, and A. Montanari, Ann. Probab. 49, 1 (2021).
  44. B. McKenna, in Annales de l'Institut Henri Poincare (B) Probabilites et statistiques, Vol. 60 (1) (Institut Henri Poincaré, 2024), pp. 636–657.
  45. V. Ros, G. Ben Arous, G. Biroli, and C. Cammarota, Phys. Rev. X 9, 011003 (2019).
  46. V. Ros, G. Biroli, and C. Cammarota, Europhys. Lett. 126, 20003 (2019).
  47. J. Kent-Dobias, Europhys. Lett. 143, 61003 (2023).
  48. J. Kent-Dobias, Phys. Rev. E 110, 064148 (2024).
  49. J. Kent-Dobias and J. Kurchan, Phys. Rev. E 107, 064111 (2023).
  50. L. Zdeborová and F. Krzakala, Adv. Phys. 65, 453 (2016).
  51. J. Barbier, arXiv:2010.14863.
  52. D. Gamarnik, C. Moore, and L. Zdeborová, J. Stat. Mech. (2022) 114015.
  53. A. Choromanska, Y. LeCun, and G. B. Arous, in Proceedings of the 28th Conference on Learning Theory (PMLR, 2015), pp. 1756–1760.
  54. G. S. Hartnett, E. Parker, and E. Geist, Phys. Rev. E 98, 022116 (2018).
  55. A. L. Moustakas, S. H. Simon, and A. M. Sengupta, IEEE Trans. Inform. Theory 49, 2545 (2003).
  56. R. Couillet, M. Debbah, and J. W. Silverstein, IEEE Trans. Inform. Theory 57, 3493 (2011).
  57. M. Mezard and A. Montanari, Information, Physics, and Computation (Oxford University Press, Oxford, 2009).
  58. W. Hachem, O. Khorunzhiy, P. Loubaton, J. Najim, and L. Pastur, IEEE Trans. Inform. Theory 54, 3987 (2008).
  59. A. L. Moustakas and P. Kazakopoulos, IEEE Trans. Inform. Theory 59, 6490 (2013).
  60. T. Tao, Topics in Random Matrix Theory, Vol. 132 (American Mathematical Soc., Providence, 2012).
  61. D. Paul and J. W. Silverstein, J. Multivariate Anal. 100, 37 (2009).
  62. B. Bercu, F. Gamboa, and M. Lavielle, ESAIM Probab. Stat. 4, 1 (2000).
  63. A. Dembo and O. Zeitouni, Large Deviations Techniques and Applications, Vol. 38 (Springer Science & Business Media, New York, 2009).
  64. A. L. Moustakas and S. H. Simon, IEEE Trans. Inform. Theory 53, 3887 (2007).
  65. Data available at http://scholar.uoa.gr/sites/default/files/arislm/files/tm25_figdata.txt.
  66. G. Folena and F. Zamponi, SciPost Phys. 15, 109 (2023).
  67. Y. Kabashima, J. Phys.: Conf. Ser. 95, 012001 (2008).
  68. A. Maillard, B. Loureiro, F. Krzakala, and L. Zdeborová, in Advances in Neural Information Processing Systems, Vol. 33 (Curran Associates, Inc., Red Hook, NY, 2020), pp. 11071–11082.

Outline

Information

Phys. Rev. E 112, 065307– Published 8 December, 2025
  • Received 16 March 2025
  • Accepted 31 October 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation