- Physical Review Journals
All JournalsPhysics Magazine

Physical Review E

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Perspective
  • Open Access

Minimax entropy: The statistical physics of optimal models

David P. Carcamo1,2, Nicholas J. Weaver1,2, Purushottam D. Dixit3,4, and Christopher W. Lynn1,2,5,*

  • *Contact author: christopher.lynn@yale.edu
PDF

Phys. Rev. E 112, 061001 – Published 3 December, 2025

DOI: https://doi.org/10.1103/kr9x-q59y

Abstract

When constructing models of the world, we aim for optimal compressions: models that include as few details as possible while remaining as accurate as possible. But which details—or features measured in data—should we choose to include in a model? Here, using the minimum description length principle, we show that the optimal features are the ones that produce the maximum entropy model with minimum entropy, thus yielding a minimax entropy principle. We review applications, which range from machine learning to optimal models of biological networks. Naive implementations, however, are limited to systems with small numbers of states and features. We therefore require new theoretical insights and computational techniques to construct optimal compressions of high-dimensional datasets arising in large-scale experiments.

View figure in article

Physics Subject Headings (PhySH)

Collections

This article appears in the following collection:

Statistical Physics Meets Machine Learning - Machine Learning Meets Statistical Physics

The Editors of Physical Review E are pleased to present the Collection on Statistical Physics Meets Machine Learning - Machine Learning Meets Statistical Physics, highlighting research at the intersection of machine learning and statistical physics, on the occasion of the two Statistical Physics Meets Machine Learning and the two Machine Learning Meets Statistical Physics sessions at the 2025 Global Physics Summit. The Collection is being guest edited by David Schwab (CUNY, New York) and Yuhai Tu (IBM Watson Research Center Yorktown Heights, NY). Every article published in this collection underwent a rigorous peer review process, adhering to the same high standards applied to all papers. The Physical Review E editorial team managed the peer review and made all editorial decisions.

Article Text

References (81)

  1. C. E. Shannon, A mathematical theory of communication, Bell Syst. Tech. J. 27, 379 (1948).
  2. T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed. (John Wiley & Sons, New York, 2006).
  3. M. Feder, Maximum entropy as a special case of the minimum description length criterion (Corresp.), IEEE Trans. Inf. Theory 32, 847 (1986).
  4. P. D. Grünwald, The Minimum Description Length Principle (MIT Press, Cambridge, 2007).
  5. M. H. Hansen and B. Yu, Model selection and the principle of minimum description length, J. Am. Stat. Assoc. 96, 746 (2001).
  6. T. Hastie, J. Friedman, and R. Tisbshirani, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Springer Series in Statistics (Springer, New York, NY, 2017).
  7. S. C. Zhu, Y. N. Wu, and D. Mumford, Minimax entropy principle and its application to texture modeling, Neural Comput. 9, 1627 (1997).
  8. S. C. Zhu, Y. Wu, and D. Mumford, FRAME: filters, random fields, and minimax entropy towards a unified theory for texture modeling, in Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition (IEEE, Piscataway, NJ, 1996), pp. 686–693.
  9. C. W. Lynn, Q. Yu, R. Pang, W. Bialek, and S. E. Palmer, Exactly solvable statistical physics models for large neuronal populations, Phys. Rev. Res. 7, L022039 (2025).
  10. C. W. Lynn, Q. Yu, R. Pang, S. E. Palmer, and W. Bialek, Exact minimax entropy models of large-scale neuronal activity, Phys. Rev. E 111, 054411 (2025).
  11. D. P. Carcamo and C. W. Lynn, Statistical physics of large-scale neural activity with loops, Proc. Natl. Acad. Sci. USA 122, e2426926122 (2025).
  12. N. J. Weaver, J. I. Faskowitz, R. F. Betzel, and C. W. Lynn, Quantifying the compressibility of the human brain, arXiv:2510.16327.
  13. E. T. Jaynes, Information theory and statistical mechanics, Phys. Rev. 106, 620 (1957).
  14. S. Pressé, K. Ghosh, J. Lee, and K. A. Dill, Principles of maximum entropy and maximum caliber in statistical physics, Rev. Mod. Phys. 85, 1115 (2013).
  15. H. C. Nguyen, R. Zecchina, and J. Berg, Inverse statistical problems: From the inverse Ising problem to data science, Adv. Phys. 66, 197 (2017).
  16. L. Zdeborová and F. Krzakala, Statistical physics of inference: Thresholds and algorithms, Adv. Phys. 65, 453 (2016).
  17. J. Chayes, L. Chayes, and E. H. Lieb, The inverse problem in classical statistical mechanics, Commun. Math. Phys. 93, 57 (1984).
  18. S. G. Brush, History of the Lenz-Ising model, Rev. Mod. Phys. 39, 883 (1967).
  19. E. Schneidman, M. J. Berry II, R. Segev, and W. Bialek, Weak pairwise correlations imply strongly correlated network states in a neural population, Nature (London) 440, 1007 (2006).
  20. A. Ashourvan, P. Shah, A. Pines, S. Gu, C. W. Lynn, D. S. Bassett, K. A. Davis, and B. Litt, Pairwise maximum entropy model explains the role of white matter structure in shaping emergent co-activation states, Commun. Biol. 4, 210 (2021) .
  21. R. E. Rosch, D. R. W. Burrows, C. W. Lynn, and A. Ashourvan, Spontaneous brain activity emerges from pairwise interactions in the larval zebrafish brain, Phys. Rev. X 14, 031050 (2024).
  22. L. Meshulam, J. L. Gauthier, C. D. Brody, D. W. Tank, and W. Bialek, Collective behavior of place and non-place neurons in the hippocampal network, Neuron 96, 1178 (2017).
  23. G. Tkačik, O. Marre, D. Amodei, E. Schneidman, W. Bialek, and M. J. Berry II, Searching for collective behavior in a large network of sensory neurons, PLoS Comput. Biol. 10, e1003408 (2014).
  24. O. Marre, S. El Boustani, Y. Frégnac, and A. Destexhe, Prediction of spatiotemporal patterns of neural activity from pairwise correlations, Phys. Rev. Lett. 102, 138101 (2009).
  25. D. S. Marks, L. J. Colwell, R. Sheridan, T. A. Hopf, A. Pagnani, R. Zecchina, and C. Sander, Protein 3D structure computed from evolutionary sequence variation, PLoS ONE 6, e28766 (2011).
  26. M. Weigt, R. A. White, H. Szurmant, J. A. Hoch, and T. Hwa, Identification of direct residue contacts in protein–protein interaction by message passing, Proc. Natl. Acad. Sci. USA 106, 67 (2009).
  27. W. P. Russ, M. Figliuzzi, C. Stocker, P. Barrat-Charlaix, M. Socolich, P. Kast, D. Hilvert, R. Monasson, S. Cocco, M. Weigt et al., An evolution-based model for designing chorismate mutase enzymes, Science 369, 440 (2020).
  28. F. Morcos, A. Pagnani, B. Lunt, A. Bertolino, D. S. Marks, C. Sander, R. Zecchina, J. N. Onuchic, T. Hwa, and M. Weigt, Direct-coupling analysis of residue coevolution captures native contacts across many protein families, Proc. Natl. Acad. Sci. USA 108, E1293 (2011).
  29. T. R. Lezon, J. R. Banavar, M. Cieplak, A. Maritan, and N. V. Fedoroff, Using the principle of entropy maximization to infer genetic interaction networks from gene expression patterns, Proc. Natl. Acad. Sci. USA 103, 19033 (2006).
  30. P. D. Dixit, Quantifying extrinsic noise in gene expression using the maximum entropy framework, Biophys. J. 104, 2743 (2013).
  31. M. Di Pierro, B. Zhang, E. L. Aiden, P. G. Wolynes, and J. N. Onuchic, Transferable model for chromosome architecture, Proc. Natl. Acad. Sci. USA 113, 12168 (2016).
  32. X. Lin, Y. Qi, A. P. Latham, and B. Zhang, Multiscale modeling of genome organization with maximum entropy optimization, J. Chem. Phys. 155, 010901 (2021).
  33. G. Shi and D. Thirumalai, A maximum-entropy model to predict 3D structural ensembles of chromatin from pairwise distances with applications to interphase chromosomes and structural variants, Nat. Commun. 14, 1150 (2023).
  34. J. J. Messelink, M. C. F. van Teeseling, J. Janssen, M. Thanbichler, and C. P. Broedersz, Learning the distribution of single-cell chromosome conformations in bacteria reveals emergent order across genomic scales, Nat. Commun. 12, 1963 (2021).
  35. P. Farré and E. Emberly, A maximum-entropy model for predicting chromatin contacts, PLoS Comput. Biol. 14, e1005956 (2018).
  36. W. Bialek, A. Cavagna, I. Giardina, T. Mora, E. Silvestri, M. Viale, and A. M. Walczak, Statistical mechanics for natural flocks of birds, Proc. Natl. Acad. Sci. USA 109, 4786 (2012).
  37. A. Cavagna, I. Giardina, F. Ginelli, T. Mora, D. Piovani, R. Tavarone, and A. M. Walczak, Dynamical maximum entropy approach to flocking, Phys. Rev. E 89, 042707 (2014).
  38. W. Bialek, A. Cavagna, I. Giardina, T. Mora, O. Pohl, E. Silvestri, M. Viale, and A. M. Walczak, Social interactions dominate speed control in poising natural flocks near criticality, Proc. Natl. Acad. Sci. USA 111, 7212 (2014).
  39. C. W. Lynn, L. Papadopoulos, D. D. Lee, and D. S. Bassett, Surges of collective human activity emerge from simple pairwise correlations, Phys. Rev. X 9, 011022 (2019).
  40. E. Schneidman, S. Still, M. J. Berry, and W. Bialek, Network information and connected correlations, Phys. Rev. Lett. 91, 238701 (2003).
  41. J. P. Sethna, Statistical Mechanics: Entropy, Order Parameters, and Complexity (Oxford University Press, USA, 2021), Vol. 14.
  42. A. M. Ferrenberg and D. P. Landau, Critical behavior of the three-dimensional Ising model: A high-resolution Monte Carlo study, Phys. Rev. B 44, 5081 (1991).
  43. R. G. Parker and R. L. Rardin, Discrete Optimization (Elsevier, Amsterdam, 2014).
  44. D. Jungnickel, Graphs, Networks and Algorithms, Algorithms and Computation in Mathematics (Springer, Berlin, Heidelberg, 2005), Vol. 3.
  45. D. Edwards, Introduction to Graphical Modelling, Springer Texts in Statistics (Springer, New York, NY, 2012).
  46. M. Yuan and Y. Lin, Model selection and estimation in the Gaussian graphical model, Biometrika 94, 19 (2007).
  47. S. Epskamp, L. J. Waldorp, R. Mõttus, and D. Borsboom, The Gaussian graphical model in cross-sectional and time-series data, Multivar. Behav. Res. 53, 453 (2018).
  48. T. Wang, Z. Ren, Y. Ding, Z. Fang, Z. Sun, M. L. MacDonald, R. A. Sweet, J. Wang, and W. Chen, FastGGM: An efficient algorithm for the inference of Gaussian graphical model in biological networks, PLoS Comput. Biol. 12, e1004755 (2016).
  49. C. Uhler, Gaussian graphical models: An algebraic and geometric perspective, arXiv:1707.04345.
  50. D. C. Van Essen, S. M. Smith, D. M. Barch, T. E. J. Behrens, E. Yacoub, K. Ugurbil, The WU-Minn Human Connectome Project: An overview, Neuroimage 80, 62 (2013).
  51. D. S. Bassett and O. Sporns, Network neuroscience, Nat. Neurosci. 20, 353 (2017).
  52. C. W. Lynn and D. S. Bassett, The physics of brain network structure, function and control, Nat. Rev. Phys. 1, 318 (2019).
  53. T. Mora, A. M. Walczak, W. Bialek, and C. G. Callan Jr., Maximum entropy models for antibody diversity, Proc. Natl. Acad. Sci. USA 107, 5405 (2010).
  54. C. Stringer, M. Pachitariu, N. Steinmetz, M. Carandini, and K. D. Harris, High-dimensional geometry of population responses in visual cortex, Nature (London) 571, 361 (2019).
  55. R. J. Baxter, Exactly Solved Models in Statistical Mechanics (Elsevier, Amsterdam, 2016).
  56. C. Chow and C. Liu, Approximating discrete probability distributions with dependence trees, IEEE Trans. Inf. Theory 14, 462 (1968).
  57. C. Moore and S. Mertens, The Nature of Computation (Oxford University Press, Oxford, 2011).
  58. O. J. Rosten, Fundamentals of the exact renormalization group, Phys. Rep. 511, 177 (2012).
  59. C. Stringer, M. Pachitariu, N. Steinmetz, C. B. Reddy, M. Carandini, and K. D. Harris, Spontaneous behaviors drive multidimensional, brainwide activity, Science 364, eaav7893 (2019).
  60. L. A. David, A. C. Materna, J. Friedman, M. I. Campos-Baptista, M. C. Blackburn, A. Perrotta, S. E. Erdman, and E. J. Alm, Host lifestyle affects human microbiota on daily timescales, Genome Biol. 15, R89 (2014).
  61. H. Rowley, S. Baluja, and T. Kanade, Neural network-based face detection, IEEE Trans. Pattern Anal. Machine Intell. 20, 23 (1998).
  62. Y. Zhang, R. Jin, and Z.-H. Zhou, Understanding bag-of-words model: a statistical framework, Int. J. Mach. Learn. Cybern. 1, 43 (2010).
  63. L. Deng, The MNIST database of handwritten digit images for machine learning research, IEEE Signal Proc. Mag. 29, 141 (2012).
  64. P. D. Dixit, Thermodynamic inference of data manifolds, Phys. Rev. Res. 2, 023201 (2020).
  65. S. J. Phillips, M. Dudík, and R. E. Schapire, A maximum entropy approach to species distribution modeling, in Proceedings of the Twenty-First International Conference on Machine Learning (ACM, New York, 2004), p. 83.
  66. A. Ratnaparkhi, Learning to parse natural language with maximum entropy models, Mach. Learn. 34, 151 (1999).
  67. D. H. de Groot, S. X. M. Leonardo, M. Pachkov, and E. van Nimwegen, Bonsai: Tree representations for distortion-free visualization and exploratory analysis of single-cell omics data, bioRxiv:2025.05.08.652944 (2025).
  68. D. J. Thouless, P. W. Anderson, and R. G. Palmer, Solution of “Solvable model of a spin glass”, Philos. Mag.: J. Theor. Expt. Appl. Phys. 35, 593 (1977).
  69. T. Tanaka, Mean-field theory of Boltzmann machine learning, Phys. Rev. E 58, 2302 (1998).
  70. Y. Roudi, J. Tyrcha, and J. Hertz, Ising model for neural data: Model quality and approximate methods for extracting functional connectivity, Phys. Rev. E 79, 051915 (2009).
  71. J. M. Beggs and D. Plenz, Neuronal avalanches in neocortical circuits, J. Neurosci. 23, 11167 (2003).
  72. G. Tkačik, T. Mora, O. Marre, D. Amodei, S. E. Palmer, M. J. Berry, and W. Bialek, Thermodynamics and signatures of criticality in a network of neurons, Proc. Natl. Acad. Sci. USA 112, 11508 (2015).
  73. L. Meshulam, J. L. Gauthier, C. D. Brody, D. W. Tank, and W. Bialek, Coarse graining, fixed points, and scaling in a large population of neurons, Phys. Rev. Lett. 123, 178103 (2019).
  74. L. Di Carlo, F. Mignacco, C. W. Lynn, and W. Bialek, Extended mean-field theories for networks of real neurons, arXiv:2504.15197.
  75. P. Ravikumar, M. J. Wainwright, and J. D. Lafferty, High-dimensional Ising model selection using ℓ1-regularized logistic regression, Ann. Stat. 38, 1287 (2010).
  76. E. Aurell and M. Ekeberg, Inverse Ising inference using all the data, Phys. Rev. Lett. 108, 090201 (2012).
  77. V. K. Olsen, J. R. Whitlock, and Y. Roudi, The quality and complexity of pairwise maximum entropy models for large cortical populations, PLoS Comput. Biol. 20, e1012074 (2024).
  78. S. Cocco and R. Monasson, Adaptive cluster expansion for the inverse Ising problem: Convergence, algorithm and tests, J. Stat. Phys. 147, 252 (2012).
  79. S. Fujishige, Submodular Functions and Optimization, Annals of Discrete Mathematics (Elsevier, Amsterdam, 2005), Vol. 58.
  80. A. Krause and D. Golovin, Submodular function maximization, Tractability 3, 3 (2014).
  81. M. Bauer, M. D. Petkova, T. Gregor, E. F. Wieschaus, and W. Bialek, Trading bits in the readout from a genetic network, Proc. Natl. Acad. Sci. USA 118, e2109011118 (2021).

Outline

Information

Phys. Rev. E 112, 061001– Published 3 December, 2025
  • Received 2 May 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation