- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Statistical mechanics of dynamical system identification

Andrei A. Klishin1,2,3,*, Joseph Bakarji2,3,4, J. Nathan Kutz2,5, and Krithika Manohar2,3,†

  • *Contact author: aklishin@hawaii.edu
  • †Contact author: kmanohar@uw.edu
PDF

Phys. Rev. Research 7, 033181 – Published 21 August, 2025

DOI: https://doi.org/10.1103/4d98-tdlp

Abstract

Recovering dynamical equations from observed noisy data is the central challenge of system identification. We develop a statistical mechanics approach to analyze sparse equation discovery algorithms, which typically balance data fit and parsimony via hyperparameter tuning. In this framework, statistical mechanics offers tools to analyze the interplay between complexity and fitness similarly to that of entropy and energy in physical systems. To establish this analogy, we define the hyperparameter optimization procedure as a two-level Bayesian inference problem that separates variable selection from coefficient inference and enables the computation of the posterior parameter distribution in closed form. Our approach provides uncertainty quantification, crucial in the low-data limit that is frequently encountered in real-world applications. A key advantage of employing statistical mechanical concepts, such as free energy and the partition function, is to connect the large-data limit to thermodynamic limit and characterize the sparsity- and noise-induced phase transitions that delineate correct from incorrect identification. We thus provide a method for closed-loop inference, estimating the noise in a given model and checking if the model is tolerant to that noise amount. This perspective of sparse equation discovery is versatile and can be adapted to various other equation discovery algorithms.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (107)

  1. L. Ljung, Perspectives on system identification, Annu. Rev. Control 34, 1 (2010).
  2. J. S. North, C. K. Wikle, and E. M. Schliep, A review of data-driven discovery for dynamic systems, Int. Stat. Rev. 91, 464 (2023).
  3. Z. Li, N. B. Kovachki, K. Azizzadenesheli, K. Bhattacharya, A. Stuart, A. Anandkumar et al., Fourier neural operator for parametric partial differential equations, in International Conference on Learning Representations (ICLR, Vienna, Austria, 2021).
  4. C. Rudin, C. Chen, Z. Chen, H. Huang, L. Semenova, and C. Zhong, Interpretable machine learning: Fundamental principles and 10 grand challenges, Stat. Surv. 16, 1 (2022).
  5. Z. C. Lipton, The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery, Queue 16, 31 (2018).
  6. C. Rudin, Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, Nat. Mach. Intell. 1, 206 (2019).
  7. J. P. Crutchfield and B. S. McNamara, Equations of motion from a data series, Complex Syst. 1, 417 (1987).
  8. S. L. Brunton, J. L. Proctor, and J. N. Kutz, Discovering governing equations from data by sparse identification of nonlinear dynamical systems, Proc. Natl. Acad. Sci. USA 113, 3932 (2016).
  9. S. H. Rudy, S. L. Brunton, J. L. Proctor, and J. N. Kutz, Data-driven discovery of partial differential equations, Sci. Adv. 3, e1602614 (2017).
  10. E. Kaiser, J. N. Kutz, and S. L. Brunton, Sparse identification of nonlinear dynamics for model predictive control in the low-data limit, Proc. R. Soc. A 474, 20180335 (2018).
  11. D. A. Messenger and D. M. Bortz, Weak SINDy: Galerkin-based data-driven model selection, Multiscale Model. Simul. 19, 1474 (2021).
  12. J. M. Hokanson, G. Iaccarino, and A. Doostan, Simultaneous identification and denoising of dynamical systems, SIAM J. Sci. Comput. 45, A1413 (2023).
  13. M. Schmidt and H. Lipson, Distilling free-form natural laws from experimental data, Science 324, 81 (2009).
  14. S.-M. Udrescu, A. Tan, J. Feng, O. Neto, T. Wu, and M. Tegmark, AI Feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity, Adv. Neural Inf. Process. Syst. 33, 4860 (2020).
  15. B. C. Daniels and I. Nemenman, Automated adaptive inference of phenomenological dynamical models, Nat. Commun. 6, 8133 (2015).
  16. S. Sahoo, C. Lampert, and G. Martius, Learning equations for extrapolation and control, in International Conference on Machine Learning (PMLR, 2018), pp. 4442–4450.
  17. N. M. Mangan, S. L. Brunton, J. L. Proctor, and J. N. Kutz, Inferring biological networks by sparse identification of nonlinear dynamics, IEEE Trans. Mol. Biol. Multi-Scale Commun. 2, 52 (2016).
  18. X. Yang, L. Guo, Z. Zheng, N. Riemer, and C. W. Tessum, Atmospheric chemistry surrogate modeling with sparse identification of nonlinear dynamics, J. Geophys. Res.: Mach. Learn. Comput. 1, e2024JH000132 (2024).
  19. J. Bakarji and D. M. Tartakovsky, Data-driven discovery of coarse-grained equations, J. Comput. Phys. 434, 110219 (2021).
  20. R. Supekar, B. Song, A. Hastewell, G. P. Choi, A. Mietke, and J. Dunkel, Learning hydrodynamic equations for active matter from particle simulations and experiments, Proc. Natl. Acad. Sci. USA 120, e2206994120 (2023).
  21. J.-C. Loiseau and S. L. Brunton, Constrained sparse galerkin regression, J. Fluid Mech. 838, 42 (2018).
  22. J. N. Kutz and S. L. Brunton, Parsimony as the ultimate regularizer for physics-informed machine learning, Nonlin. Dyn. 107, 1801 (2022).
  23. F. Van Breugel, J. N. Kutz, and B. W. Brunton, Numerical differentiation of noisy data: A unifying multi-objective optimization framework, IEEE Access 8, 196865 (2020).
  24. M. L. Gao and J. N. Kutz, Bayesian autoencoders for data-driven discovery of coordinates, governing equations and fundamental constants, Proc. R. Soc. A 480, 20230506 (2024).
  25. J. S. North, C. K. Wikle, and E. M. Schliep, A Bayesian approach for data-driven dynamic equation discovery, J. Agric. Biol. Environ. Stat. 27, 728 (2022).
  26. L. Fung, U. Fasel, and M. Juniper, Rapid Bayesian identification of sparse nonlinear dynamics from scarce and noisy data, Proc. R. Soc. A. 481, 20240200 (2025).
  27. J. Wentz and A. Doostan, Derivative-based SINDy (DSINDy): Addressing the challenge of discovering governing equations from noisy data, Comput. Methods Appl. Mech. Eng. 413, 116096 (2023).
  28. P. Langley and S. Sage, Tractable average-case analysis of naive Bayesian classifiers, in ICML, Vol. 99 (Citeseer, Bled, Slovenia, 1999), pp. 220–228.
  29. S. Ganguli and H. Sompolinsky, Statistical mechanics of compressed sensing, Phys. Rev. Lett. 104, 188701 (2010).
  30. R. R. Nadakuditi and M. E. Newman, Graph spectra and the detectability of community structure in networks, Phys. Rev. Lett. 108, 188701 (2012).
  31. L. Zdeborová and F. Krzakala, Statistical physics of inference: Thresholds and algorithms, Adv. Phys. 65, 453 (2016).
  32. T. Obuchi, Y. Nakanishi-Ohno, M. Okada, and Y. Kabashima, Statistical mechanical analysis of sparse linear regression as a variable selection problem, J. Stat. Mech. (2018) 103401.
  33. F. Krzakala and L. Zdeborová, Statistical physics methods in optimization and machine learning: An introduction to replica, cavity & message-passing techniques (2022).
  34. D. Bertsimas and W. Gurnee, Learning sparse nonlinear dynamics via mixed-integer optimization, Nonlin. Dyn. 111, 6585 (2023).
  35. S. M. Hirsh, D. A. Barajas-Solano, and J. N. Kutz, Sparsifying priors for Bayesian uncertainty quantification in model discovery, R. Soc. Open Sci. 9, 211823 (2022).
  36. Z. Wang, B. Wu, K. Garikipati, and X. Huan, A perspective on regression and Bayesian approaches for system identification of pattern formation dynamics, Theor. Appl. Mech. Lett. 10, 188 (2020).
  37. Y. C. Mathpati, T. Tripura, R. Nayek, and S. Chakraborty, Discovering stochastic partial differential equations from limited data using variational Bayes inference, Comput. Methods Appl. Mech. Eng. 418, 116512 (2024).
  38. T. Tripura and S. Chakraborty, A sparse Bayesian framework for discovering interpretable nonlinear stochastic dynamical systems with Gaussian white noise, Mech. Syst. Signal Process. 187, 109939 (2023).
  39. R. Nayek, R. Fuentes, K. Worden, and E. J. Cross, On spike-and-slab priors for Bayesian equation discovery of nonlinear dynamical systems via sparse linear regression, Mech. Syst. Signal Proc. 161, 107986 (2021).
  40. K. S. More, T. Tripura, R. Nayek, and S. Chakraborty, A Bayesian framework for learning governing partial differential equation from data, Physica D 456, 133927 (2023).
  41. D. Long, W. Xing, A. Krishnapriyan, R. Kirby, S. Zhe, and M. W. Mahoney, Equation discovery with Bayesian spike-and-slab priors and efficient kernels, in International Conference on Artificial Intelligence and Statistics (PMLR, 2024), pp. 2413–2421.
  42. R. K. Niven, A. Mohammad-Djafari, L. Cordier, M. Abel, and M. Quade, Bayesian identification of dynamical systems, in Proceedings, Vol. 33 (MDPI, Garching, Germany, 2020), p. 33.
  43. R. K. Niven, L. Cordier, A. Mohammad-Djafari, M. Abel, and M. Quade, Dynamical system identification, model selection, and model uncertainty quantification by Bayesian inference, Chaos 34, 083140 (2024).
  44. D. A. Messenger and D. M. Bortz, Asymptotic consistency of the wsindy algorithm in the limit of continuum data, IMA J. Numer. Anal. drae086 (2024).
  45. L. Zhang and H. Schaeffer, On the convergence of the sindy algorithm, Multiscale Model. Simul. 17, 948 (2019).
  46. G. Tran and R. Ward, Exact recovery of chaotic systems from highly corrupted data, Multiscale Model. Simul. 15, 1108 (2017).
  47. H. Schaeffer, G. Tran, and R. Ward, Extracting sparse high-dimensional dynamics from limited data, SIAM J. Appl. Math. 78, 3279 (2018).
  48. H. Schaeffer, G. Tran, R. Ward, and L. Zhang, Extracting structured dynamical systems using sparse optimization with very few samples, Multiscale Model. Simul. 18, 1435 (2020).
  49. S. Zhang and G. Lin, Robust data-driven discovery of governing physical laws with error bars, Proc. R. Soc. A 474, 20180305 (2018).
  50. U. Fasel, J. N. Kutz, B. W. Brunton, and S. L. Brunton, Ensemble-SINDy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control, Proc. R. Soc. A 478, 20210904 (2022).
  51. M. L. Gao, U. Fasel, S. L. Brunton, and J. N. Kutz, Convergence of uncertainty estimates in ensemble and Bayesian sparse model discovery, arXiv:2301.12649.
  52. D. L. Donoho, For most large underdetermined systems of linear equations the minimal L1-norm solution is also the sparsest solution, Commun. Pure Appl. Math. 59, 797 (2006).
  53. C. M. Carvalho, N. G. Polson, and J. G. Scott, Handling sparsity via the horseshoe, in Artificial Intelligence and Statistics (PMLR, 2009), pp. 73–80.
  54. A. Bereyhi, Statistical mechanics of regularized least squares, Ph.D. thesis, Friedrich-Alexander-Universität Erlangen-Nürnberg, 2020.
  55. See Supplemental Material at http://link.aps.org/supplemental/10.1103/4d98-tdlp for detailed derivations, supplementary results, and computational time benchmarks.
  56. D. J. C. Mackay, Bayesian Methods for Adaptive Models (California Institute of Technology, Pasadena, 1992).
  57. N. Goldenfeld, Lectures on Phase Transitions and the Renormalization Group (Addison-Wesley, Reading MA, 1992).
  58. H. Akaike, A new look at the statistical model identification, IEEE Trans. Autom. Control 19, 716 (1974).
  59. G. Schwarz, Estimating the dimension of a model, Ann. Stat. 6, 461 (1978).
  60. We thank L. Fung and M. Juniper for directing our attention to the natural sparsity effect which they term “Occam factor” in their paper [26].
  61. P. Zhao and B. Yu, On model selection consistency of lasso, J. Mach. Learn. Res. 7, 2541 (2006).
  62. E. N. Lorenz, Deterministic nonperiodic flow, J. Atmos. Sci. 20, 130 (1963).
  63. J. Sokol, The hidden heroines of chaos, Quanta Mag. (2019).
  64. W. Gilpin, Chaos as an interpretable benchmark for forecasting and data-driven modelling, in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) (2021).
  65. V. I. Zubov, Methods of AM Lyapunov and Their Application, Vol. 4439 (US Atomic Energy Commission, Germantown, MD, 1961).
  66. A. A. Kaptanoglu, L. Zhang, Z. G. Nicolaou, U. Fasel, and S. L. Brunton, Benchmarking sparse system identification with low-dimensional chaos, Nonlin. Dynam. 11, 13143 (2023).
  67. M. C. Cross and P. C. Hohenberg, Pattern formation outside of equilibrium, Rev. Mod. Phys. 65, 851 (1993).
  68. B. de Silva, K. Champion, M. Quade, J.-C. Loiseau, J. Kutz, and S. Brunton, PySINDy: A Python package for the sparse identification of nonlinear dynamical systems from data, J. Open Source Softw. 5, 2104 (2020).
  69. O. Rossler, An equation for hyperchaos, Phys. Lett. A 71, 155 (1979).
  70. C. Letellier and O. E. Rossler, Hyperchaos, Scholarpedia 2, 1936 (2007).
  71. A. Abooee, H. Yaghini-Bonabi, and M. R. Jahed-Motlagh, Analysis and circuitry realization of a novel three-dimensional chaotic system, Commun. Nonlin. Sci. Numer. Simul. 18, 1235 (2013).
  72. X. Hu, V. Tritschler, S. Pirozzoli, and N. Adams, Dispersion-dissipation condition for finite difference schemes, arXiv:1204.5088.
  73. K. Wu and D. Xiu, Numerical aspects for approximating governing equations using data, J. Comput. Phys. 384, 200 (2019).
  74. B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, Least angle regression, Ann. Stat. 32, 407 (2004).
  75. C. B. Delahunt and J. N. Kutz, A toolkit for data-driven discovery of governing equations in high-noise regimes, IEEE Access 10, 31210 (2022).
  76. A. Cortiella, K.-C. Park, and A. Doostan, A priori denoising strategies for sparse identification of nonlinear dynamical systems: A comparative study, J. Comput. Inf. Sci. Eng. 23, 011004 (2023).
  77. A. Lotka, Elements of Physical Biology (Williams and Wilkins, Philadelphia, PA (1925).
  78. V. Volterra, Variazioni e Fluttuazioni Del Numero D'individui in Specie Animali Conviventi (Società anonima tipografica “Leonardo da Vinci,”, Città di Castello, 1926).
  79. E. Leigh, The ecological role of Volterra's equations, in Some Mathematical Problems in Biology, edited by M. Gerstenhaber (American Mathematical Society, Washington, DC, 1968), pp. 1–61.
  80. E. P. Odum, Fundamentals of Ecology, 3rd ed. (W B Saunders, Philadelphia, PA, 1971).
  81. We use the tabulation of the Leigh and Odom datasets provided via personal communication by B. Deng [89].
  82. P. Howard, , https://people.tamu.edu/∼phoward//M442.html.
  83. J. M. Mahaffy, Lotka-Volterra models (2010), https://jmahaffy.sdsu.edu/courses/f09/math636/lectures/lotka/qualde2.html.
  84. B. Carpenter, Predator-prey population dynamics: the Lotka-Volterra model in stan, https://mc-stan.org/users/documentation/case-studies/lotka-volterra-predator-prey.html (2018).
  85. U. Wilensky and K. Reisman, Thinking like a wolf, a sheep, or a firefly: Learning biology through constructing and testing computational theories—an embodied modeling approach, Cogn. Instruct. 24, 171 (2006).
  86. W. K. Purves, G. H. Orians, and H. C. Heller, Life, the Science of Biology (Sinauer Associates, Inc., Sunderland, 1992).
  87. Carpenter [84] traces the origin of the numerical table as follows: “Howard (personal communication) traces the numerical data to Joseph M. Mahaffy's (2010) lecture notes on mathematical modeling for his San Diego State University course (Math 636) [83], which in turn cites Purves, Sadava, Orians, and Heller's book, Life: The Science of Biology [86]” (references ours).
  88. M. E. Gilpin, Do hares eat lynx? Am. Nat. 107, 727 (1973).
  89. B. Deng, An inverse problem: Trappers drove hares to eat lynx, Acta Biotheoret. 66, 213 (2018).
  90. P. A. Regalia, An unbiased equation error identifier and reduced-order approximations, IEEE Trans. Signal Process. 42, 1397 (1994).
  91. S. Rudy, A. Alla, S. L. Brunton, and J. N. Kutz, Data-driven identification of parametric partial differential equations, SIAM J. Appl. Dyn. Syst. 18, 643 (2019).
  92. M. Tang, W. Liao, R. Kuske, and S. H. Kang, Weakident: Weak formulation for identifying differential equation using narrow-fit and trimming, J. Comput. Phys. 483, 112069 (2023).
  93. H. Chen, Data-driven sparse identification of nonlinear dynamical systems using linear multistep methods, Calcolo 60, 11 (2023).
  94. D. M. Bortz, D. A. Messenger, and V. Dukic, Direct estimation of parameters in ode models using wendy: Weak-form estimation of nonlinear dynamics, Bull. Math. Biol. 85, 110 (2023).
  95. P. Gaspard and X.-J. Wang, Noise, chaos, and (ɛ, τ)-entropy per unit time, Phys. Rep. 235, 291 (1993).
  96. K. Champion, B. Lusch, J. N. Kutz, and S. L. Brunton, Data-driven discovery of coordinates and governing equations, Proc. Natl. Acad. Sci. USA 116, 22445 (2019).
  97. J. Bakarji, K. Champion, J. Nathan Kutz, and S. L. Brunton, Discovering governing equations from partial measurements with deep delay autoencoders, Proc. R. Soc. A 479, 20230422 (2023).
  98. Z. G. Nicolaou, G. Huo, Y. Chen, S. L. Brunton, and J. N. Kutz, Data-driven discovery and extrapolation of parameterized pattern-forming dynamics, Phys. Rev. Res. 5, L042017 (2023).
  99. A. A. Kaptanoglu, J. L. Callaham, A. Aravkin, C. J. Hansen, and S. L. Brunton, Promoting global stability in data-driven models of quadratic nonlinear dynamics, Phys. Rev. Fluids 6, 094401 (2021).
  100. M. Peng, A. Kaptanoglu, C. Hansen, K. Manohar, and S. Brunton, Local stability guarantees for data-driven quadratically nonlinear models, arXiv:2403.00324.
  101. R. Chartrand, Numerical differentiation of noisy, nonsmooth data, Int. Scholar. Res. Notices 2011 (2011).
  102. R. W. Schafer, What is a Savitzky-Golay filter?[lecture notes], IEEE Signal Proc. Mag. 28, 111 (2011).
  103. F. Van Breugel, Y. Liu, B. W. Brunton, and J. N. Kutz, PyNumDiff: A python package for numerical differentiation of noisy time-series data, J. Open Source Softw. 7, 4078 (2022).
  104. D. P. Foster and E. I. George, The risk inflation criterion for multiple regression, Ann. Stat. 22, 1947 (1994).
  105. D. Bertsimas and B. V. Parys, Sparse high-dimensional regression: Exact scalable algorithms and phase transitions, Ann. Stat. 48, 300 (2020).
  106. J. Bakarji and A. A. Klishin, ZSINDy (2024), https://github.com/josephbakarji/zsindy.
  107. F. Crameri, Scientific colour maps (2018), https://doi.org/10.5281/zenodo.1243862.

Outline

Information

Phys. Rev. Research 7, 033181– Published 21 August, 2025
  • Received 8 May 2024
  • Accepted 8 July 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation