- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Generalized aliasing explains double descent and informs model design

Mark K. Transtrum* and Gus L. W. Hart

Tyler J. Jarvis and Jared P. Whitehead

  • *Contact author: mktranstrum@gmail.com
PDF

Phys. Rev. Research 7, 043268 – Published 8 December, 2025

DOI: https://doi.org/10.1103/qy5r-p5b7

Abstract

A central problem in data science is to use potentially noisy samples of an unknown function to predict function values for unseen inputs. In classical statistics, the predictive error is understood as a trade-off between the bias and the variance that balances model simplicity with its ability to fit complex functions. However, overparametrized models exhibit counterintuitive behaviors, such as “double descent” in which models of increasing complexity exhibit decreasing generalization error. Other models may exhibit more complicated patterns of predictive error with multiple peaks and valleys. Neither double descent nor multiple descent phenomena are well explained by the bias-variance decomposition. We introduce a decomposition that we call the generalized aliasing decomposition (GAD) to explain the relationship between predictive performance and model complexity. The GAD decomposes the predictive error into three parts: (1) model insufficiency, which dominates when the number of parameters is much smaller than the number of data points, (2) data insufficiency, which dominates when the number of parameters is much greater than the number of data points, and (3) generalized aliasing, which dominates between these two extremes. We demonstrate the applicability of the GAD to diverse applications, including random feature models from machine learning, Fourier transforms from signal processing, solution methods for differential equations, and predictive formation enthalpy in materials discovery. Because key components of the generalized aliasing decomposition can be explicitly calculated from the relationship between model class and samples without seeing any data labels, it can answer questions related to experimental design and model selection before collecting data or performing experiments. We further demonstrate this approach on several examples and discuss implications for predictive modeling and data science.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (69)

  1. N. Goldenfeld, Simple lessons from complexity, Science 284, 87 (1999).
  2. E. P. Hoel, L. Albantakis, and G. Tononi, Quantifying causal emergence shows that macro can beat micro, Proc. Natl. Acad. Sci. USA 110, 19790 (2013).
  3. J. P. Crutchfield, The dreams of theory, WIREs Comput. Stat. 6, 75 (2014).
  4. M. K. Transtrum, B. B. Machta, K. S. Brown, B. C. Daniels, C. R. Myers, and J. P. Sethna, Perspective: Sloppiness and emergent theories in physics, biology, and beyond, J. Chem. Phys. 143, 010901 (2015).
  5. H. H. Mattingly, M. K. Transtrum, M. C. Abbott, and B. B. Machta, Maximizing the information learned from finite data selects a simple model, Proc. Natl. Acad. Sci. USA 115, 1760 (2018).
  6. P. Chvykov and E. Hoel, Causal geometry, Entropy 23, 24 (2021).
  7. K. N. Quinn, M. C. Abbott, M. K. Transtrum, B. B. Machta, and J. P. Sethna, Information geometry for multiparameter models: new perspectives on the origin of simplicity, Rep. Prog. Phys. 86, 035901 (2023).
  8. M. Belkin, D. J. Hsu, S. Ma, and S. Mandal, Reconciling modern machine-learning practice and the classical bias-variance trade-off, Proc. Natl. Acad. Sci. USA 116, 15849 (2019).
  9. Z. Yang, Y. Yu, C. You, J. Steinhardt, and Y. Ma, Rethinking bias-variance trade-off for generalization of neural networks, in International Conference on Machine Learning (PMLR, 2020), pp. 10767–10777.
  10. Y. Dar, V. Muthukumar, and R. G. Baraniuk, A farewell to the bias-variance tradeoff? An overview of the theory of overparameterized machine learning, arXiv:2109.02355.
  11. S. Geman, E. Bienenstock, and R. Doursat, Neural networks and the bias/variance dilemma, Neural Comput. 4, 1 (1992).
  12. T. Hastie, A. Montanari, S. Rosset, and R. J. Tibshirani, Surprises in high-dimensional ridgeless least squares interpolation, Ann. Stat. 50, 949 (2022).
  13. M. Loog, T. Viering, A. Mey, J. H. Krijthe, and D. M. Tax, A brief prehistory of double descent, Proc. Natl. Acad. Sci. USA 117, 10625 (2020).
  14. L. Chen, Y. Min, M. Belkin, and A. Karbasi, Multiple descent: Design your own generalization curve, in Advances in Neural Information Processing Systems, edited by M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Curran Associates, Inc., 2021), Vol. 34, pp. 8898–8912.
  15. P. Nakkiran, G. Kaplun, Y. Bansal, T. Yang, B. Barak, and I. Sutskever, Deep double descent: Where bigger models and more data hurt, J. Stat. Mech. (2021) 124003.
  16. S. d’Ascoli, M. Refinetti, G. Biroli, and F. Krzakala, Double trouble in double descent: Bias and variance(s) in the lazy regime, in International Conference on Machine Learning (PMLR, 2020), pp. 2280–2290.
  17. B. Adlam and J. Pennington, Understanding double descent requires a fine-grained bias-variance decomposition, Adv. Neural Inf. Process. Syst. 33, 11022 (2020).
  18. E. H. Lee and V. Cherkassky, Vc theoretical explanation of double descent, arXiv:2205.15549.
  19. L. Oneto, S. Ridella, and D. Anguita, Do we really need a new theory to understand the double-descent?, in ESANN 2022 Proceedings, European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (2022).
  20. R. Schaeffer, M. Khona, Z. Robertson, A. Boopathy, K. Pistunova, J. W. Rocks, I. R. Fiete, and O. Koyejo, Double descent demystified: Identifying, interpreting & ablating the sources of a deep learning puzzle, arXiv:2303.14151.
  21. M. Lafon and A. Thomas, Understanding the double descent phenomenon in deep learning, arXiv:2403.10459.
  22. J. P. Boyd, Chebyshev and Fourier Spectral methods (Courier Corporation, Dover, Mineola, New York, 2001).
  23. H. Sedghi, S. Bengio, K. Hata, A. Madry, A. Morcos, B. Neyshabur, M. Raghu, A. Rahimi, L. Schmidt, and Y. Xiao, Identifying and understanding deep learning phenomena, in ICML 2019 Workshop (ICML, 2019).
  24. Although fitting monomials is the canonical pedagogical example, the ill-conditioning of this Vandermonde matrix makes this basis ill-suited for practical applications.
  25. J. Humpherys, T. J. Jarvis, and E. J. Evans, Foundations of Applied Mathematics: Mathematical Analysis (Society for Industrial and Applied Mathematics, Philadelphia, PA, 2017), Vol. 1, pp. xx+689.
  26. X. Sheng and T. Wang, An iterative method to compute Moore-Penrose inverse based on gradient maximal convergence rate, Filomat 27, 1269 (2013).
  27. R. M. Gower and P. Richtárik, Randomized quasi-Newton updates are linearly convergent matrix inversion algorithms, SIAM J. Matrix Anal. Appl. 38, 1380 (2017).
  28. Any real analytic function is determined by its values on a dense set, so we may limit ourselves to only considering rational points t in the interval [a,b].
  29. L. Wasserman, All of statistics, Springer Texts in Statistics: A Concise Course in Statistical Inference (Springer, New York, 2004).
  30. F. P. Gantmacher and M. G. Krein, Oscillation Matrices and Kernels and Small Vibrations of Mechanical Systems, Revised edition (AMS Chelsea Publishing, Providence, RI, 2002), translation based on the 1941 Russian original, edited and with a preface by Alex Eremenko.
  31. J. R. Bunch and C. P. Nielsen, Updating the singular value decomposition, Numer. Math. 31, 111 (1978).
  32. R. C. Thompson, The behavior of eigenvalues and singular values under perturbations of restricted rank, Linear Algebra Appl. 13, 69 (1976).
  33. J. H. Wilkinson, The Algebraic Eigenvalue Problem (Clarendon Press, Oxford, 1965).
  34. M. Rudelson and R. Vershynin, Non-asymptotic theory of random matrices: extreme singular values, in Proceedings of the International Congress of Mathematicians (Hindustan Book Agency, New Delhi, 2010), Vol. III, pp. 1576–1602.
  35. S. Mei and A. Montanari, The generalization error of random features regression: Precise asymptotics and the double descent curve, Commun. Pure Appl. Math. 75, 667 (2022).
  36. R. A. Roberts and C. T. Mullis, Digital Signal Processing (Addison-Wesley Longman Publishing Co., Inc., Reading, MA, 1987).
  37. S. A. Orszag, On the elimination of aliasing in finite-difference schemes by filtering high-wavenumber components, J. Atmos. Sci. 28, 1074 (1971).
  38. D. Lerch, O. Wieckhorst, G. L. Hart, R. W. Forcade, and S. Müller, UNCLE: A code for constructing cluster expansions for arbitrary lattices with minimal user-input, Modell. Simul. Mater. Sci. Eng. 17, 055003 (2009).
  39. J. M. Sanchez, F. Ducastelle, and D. Gratias, Generalized cluster description of multicomponent systems, Physica A 128, 334 (1984).
  40. J. Sanchez, Cluster expansions and the configurational energy of alloys, Phys. Rev. B 48, 14013 (1993).
  41. J. Sanchez, Cluster expansion and the configurational theory of alloys, Phys. Rev. B 81, 224202 (2010).
  42. P. E. A. Turchi and A. Gonis, Statics and Dynamics of Alloy Phase Transformations, NATO ASI Series, Series B, Physics (Springer Science & Business Media, 2012), Vol. 319.
  43. A. van de Walle, M. Asta, and G. Ceder, The alloy theoretic automated toolkit: A user guide, Calphad 26, 539 (2002).
  44. T. Mueller and G. Ceder, Bayesian approach to cluster expansions, Phys. Rev. B 80, 024103 (2009).
  45. M. Ångqvist, W. A. Muñoz, J. M. Rahm, E. Fransson, C. Durniak, P. Rozyczko, T. H. Rod, and P. Erhart, ICET – a Python library for constructing and sampling alloy cluster expansions, Adv. Theory Simul. 2, 1900015 (2019).
  46. A. Seko, K. Yuge, F. Oba, A. Kuwabara, and I. Tanaka, Prediction of ground-state structures and order-disorder phase transitions in II–III spinel oxides: A combined cluster-expansion method and first-principles study, Phys. Rev. B 73, 184117 (2006).
  47. The site functions ξ themselves are usually discrete Cheybschev polynomials or a Fourier basis. Any functions that form an orthonormal set over the discrete values of σi are suitable.
  48. G. L. Hart, V. Blum, M. J. Walorski, and A. Zunger, Evolutionary approach for determining first-principles hamiltonians, Nat. Mater. 4, 391 (2005).
  49. V. Blum, G. L. W. Hart, M. J. Walorski, and A. Zunger, Using genetic algorithms to map first-principles results to model Hamiltonians: Application to the generalized Ising model for alloys, Phys. Rev. B 72, 165113 (2005).
  50. A. Seko, Y. Koyama, and I. Tanaka, Cluster expansion method for multicomponent systems based on optimal selection of structures for density-functional theory calculations, Phys. Rev. B 80, 165122 (2009).
  51. L. J. Nelson, V. Ozoliņš, C. S. Reese, F. Zhou, and G. L. Hart, Cluster expansion made easy with Bayesian compressive sensing, Phys. Rev. B 88, 155105 (2013).
  52. L. J. Nelson, G. L. Hart, F. Zhou, and V. Ozoliņš, Compressive sensing as a paradigm for building physics models, Phys. Rev. B 87, 035125 (2013).
  53. J. M. Sanchez, Renormalized interactions in truncated cluster expansions, Phys. Rev. B 99, 134206 (2019).
  54. B. Puchala and A. Van der Ven, Thermodynamics of the Zr-O system from first-principles calculations, Phys. Rev. B 88, 094108 (2013).
  55. Z. Leong and T. L. Tan, Robust cluster expansion of multicomponent systems using structured sparsity, Phys. Rev. B 100, 134108 (2019).
  56. G. L. Hart and R. W. Forcade, Algorithm for generating derivative structures, Phys. Rev. B 77, 224115 (2008).
  57. G. L. Hart and R. W. Forcade, Generating derivative structures from multilattices: Algorithm and application to hcp alloys, Phys. Rev. B 80, 014120 (2009).
  58. G. L. Hart, L. J. Nelson, and R. W. Forcade, Generating derivative structures at a fixed concentration, Comput. Mater. Sci. 59, 101 (2012).
  59. Until recently, enumerating all linearly independent basis functions, without also generating many linearly dependent basis functions, was an outstanding problem. This algorithm is not yet published.
  60. These formation enthalpy data were generated by “unrelaxed” Density functional theory calculations for configurations of platinum and copper, which were then adjusted by a linear regression using slight regularization to smooth out noise.
  61. The Vandermonde matrix is well conditioned in the special case that the inputs t all lie on the unit circle U1={t∈C:|z|=1}, but it is badly conditioned if the inputs t do not have unit modulus.
  62. F. F. Yilmaz and R. Heckel, Regularization-wise double descent: Why it occurs and how to eliminate it, in 2022 IEEE International Symposium on Information Theory (ISIT) (IEEE, Piscataway, NJ, 2022), pp. 426–431.
  63. P. Nakkiran, P. Venkat, S. Kakade, and T. Ma, Optimal regularization can mitigate double descent, arXiv:2003.01897.
  64. B. B. Machta, R. Chachra, M. K. Transtrum, and J. P. Sethna, Parameter space compression underlies emergent theories and predictive models, Science 342, 604 (2013).
  65. M. K. Transtrum and P. Qiu, Model reduction by manifold boundaries, Phys. Rev. Lett. 113, 098701 (2014).
  66. D. D. Hoffman, M. Singh, and C. Prakash, The interface theory of perception, Psychon. Bull. Rev. 22, 1480 (2015).
  67. J. Lee, L. Xiao, S. S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Pennington, Wide neural networks of any depth evolve as linear models under gradient descent, J. Stat. Mech. (2020) 124002.
  68. M. K. Transtrum, B. B. Machta, and J. P. Sethna, Geometry of nonlinear least squares with applications to sloppy models and optimization, Phys. Rev. E 83, 036701 (2011).
  69. M. K. Transtrum, G. L. W. Hart, T. J. Jarvis, and J. P. Whitehead, Data and code for generalized aliasing, GitHub (2025), https://github.com/mktranstrum/eGAD.

Outline

Information

Phys. Rev. Research 7, 043268– Published 8 December, 2025
  • Received 10 January 2025
  • Accepted 12 September 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation