- Physical Review Journals
All JournalsPhysics Magazine

PRX Life

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Probabilistic Models, Compressible Interactions, and Neural Coding

Luisa Ramirez1,2,3,*, William Bialek3,4,5,†, Stephanie E. Palmer4,6,†, and David J. Schwab4,†

  • *Contact author: luramire@uni-mainz.de
  • †These three senior authors (W.B., S.E.P., and D.J.S.) contributed to all aspects of the work.
PDF

PRX Life 4, 023008 – Published 23 April, 2026

DOI: https://doi.org/10.1103/pj4s-g4jc

Abstract

In physics we often use very simple models to describe systems with many degrees of freedom, but it is not clear why or how this success can be transferred to the more complex biological context. We consider models for the joint distribution of many variables, as with the combinations of spiking and silence in large networks of neurons. In this probabilistic framework, we argue that simple models are possible if the mutual information between two halves of the system is consistently subextensive, and if this shared information is compressible. These conditions are not met generically, but they are met by real-world data, such as natural images and the activity in a population of retinal output neurons. We introduce compression strategies that combine the information bottleneck with an iteration scheme inspired by the renormalization group, and we find that the number of parameters needed to describe the distribution of joint activity scales with the square of the number of neurons, even though the interactions are not well approximated as pairwise. Our results also show that this shared information is essentially equal to the information that individual neurons carry about natural visual inputs, which has surprising implications for the neural code.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (80)

  1. J. Sethna, Statistical Mechanics: Entropy, Order Parameters, and Complexity (Oxford University Press, Oxford, UK, 2006).
  2. C. Itzykson and J. M. Drouffe, Statistical Field Theory (Cambridge University Press, Cambridge, UK, 1991).
  3. E. Lubeck and L. Cai, Single-cell systems biology by super-resolution imaging and combinatorial labeling, Nat. Methods 9, 743 (2012).
  4. A. M. Klein et al., Droplet barcoding for single cell transcriptomics applied to embryonic stem cells, Cell 161, 1187 (2015).
  5. E. Z. Macossko et al., Highly parallel genome-wide expression profiling of individual cells using nanoliter droplets, Cell 161, 1202 (2015).
  6. K. H. Chen, A. N. Boettiger, J. R. Moffitt, S. S. Wang, and X. Zhuang, Spatially resolved, highly multiplexed RNA profiling in single cells, Science 348, aaa6090 (2015).
  7. R. Segev, J. Goodhouse, J. Puchalla, and M. J. Berry II, Recording spikes from a large fraction of the ganglion cells in a retinal patch, Nat. Neurosci. 7, 1155 (2004).
  8. D. A. Dombeck, C. D. Harvey, L. Tian, L. L. Looger, and D. W. Tank, Functional imaging of hippocampal place cells at cellular resolution during virtual navigation, Nat. Neurosci. 13, 1433 (2010).
  9. O. Marre, D. Amodei, N. Deshmukh, K. Sadeghi, F. Soo, T. E. Holy, and M. J. Berry II, Mapping a complete neural population in the retina, J. Neurosci. 32, 14859 (2012).
  10. M. B. Ahrens, M. B. Orger, D. N. Robson, J. M. Li, and P. J. Keller, Whole-brain functional imaging at cellular resolution using light-sheet microscopy, Nat. Methods 10, 413 (2013).
  11. J. J. Jun et al., Fully integrated silicon probes for high-density recording of neural activity, Nature (London) 551, 232 (2017).
  12. J. E. Chung et al., High-density, long-lasting, and multi-region electrophysiological recordings using polymer electrode arrays, Neuron 101, 21 (2019).
  13. J. Demas, J. Manley, F. Tejara, K. Barber, H. Kim, F. M. Traub, B. Chen, and A. Vaziri, High-speed, cortex-wide volumetric recording of neuroactivity at cellular resolution using light beads microscopy, Nat. Methods 18, 1103 (2021).
  14. A. Cavagna and I. Giardina, Bird flocks as condensed matter, Annu. Rev. Condens. Matter Phys. 5, 183 (2014).
  15. Y. LeCun, Y. Bengio, and G. Hinton, Deep learning, Nature (London) 521, 436 (2015).
  16. P. Mehta, M. Bukov, C.-H. Wang, A. G. R. Day, C. Richardson, C. K. Fisher, and D. J. Schwab, A high-bias, low-variance introduction to machine learning for physicists, Phys. Rep. 810, 1 (2019).
  17. G. Carleo, I. Cirac, K. Cranmer, L. Daudet, M. Schuld, N. Tishby, L.Vogt-Maranto, and L. Zdeborová, Machine learning and the physical sciences, Rev. Mod. Phys. 91, 045002 (2019).
  18. B. M. Yu, J. P. Cunningham, G. Santhanam, S. I. Ryu, K. V. Shenoy, and M. Sahani, Gaussian-process factor analysis for low-dimensional single-trial analysis of neural population activity, J. Neurophysiol. 102, 614 (2009).
  19. J. A. Gallego, M. G. Perich, L. E. Miller, and S. A. Solla, Neural manifolds for the control of movement, Neuron 94, 978 (2017).
  20. M. K. Transtrum, B. B. Machta, K. S. Brown, B. C. Daniels, C. R. Myers, and J. P. Sethna, Perspective: Sloppiness and emergent theories in physics, biology, and beyond, J. Chem. Phys. 143, 010901 (2015).
  21. K. N. Quinn, H. Wilber, A. Townsend, and J. P. Sethna, Chebyshev approximation and the global geometry of model predictions, Phys. Rev. Lett. 122, 158302 (2019).
  22. E. Schneidman, M. J. Berry II, R. Segev, and W. Bialek, Weak pairwise correlations imply strongly correlated network states in a neural population, Nature (London) 440, 1007 (2006).
  23. W. Bialek and R. Ranganathan, Rediscovering the power of pairwise interactions, arXiv:0712.4397.
  24. M. Weigt, R. A. White, H. Szurmant, J. A. Hoch, and T. Hwa, Identification of direct residue contacts in protein-protein interaction by message passing, Proc. Natl. Acad. Sci. USA 106, 67 (2009).
  25. D. S. Marks, L. J. Colwell, R. Sheridan, T. A. Hopf, A. Pagnani, R. Zecchina, and C. Sander, Protein 3D structure computed from evolutionary sequence variation, PLoS ONE 6, e28766 (2011).
  26. G. Tkačik, O. Marre, D. Amodei, E. Schneidman, W. Bialek, and M. J. Berry II, Searching for collective behavior in a large network of sensory neurons, PLoS Comput. Biol. 10, e1003408 (2014).
  27. L. Meshulam, J. L. Gauthier, C. D. Brody, D. W. Tank, and W. Bialek, Collective behavior of place and non-place neurons in the hippocampal network, Neuron 96, 1178 (2017).
  28. H. C. Nguyen, R. Zecchina, and J. Berg, Inverse statistical problems: From the inverse Ising problem to data science, Adv. Phys. 66, 197 (2017).
  29. A. Cavagna, I. Giardina, and T. Grigera, The physics of flocking: Correlation as a compass from experiments to theory, Phys. Rep. 728, 1 (2018).
  30. L. Meshulam, J. L. Gauthier, C. D. Brody, D. W. Tank, and W. Bialek, Coarse-graining, fixed points, and scaling in a large population of neurons, Phys. Rev. Lett. 123, 178103 (2019).
  31. G. B. Morales, S. Di Santo, and M. A. Muñoz, Quasi-universal scaling in mouse brain neuronal activity stems from edge-of-instability critical dynamics, Proc. Natl. Acad. Sci. USA 120, e2208998120 (2023).
  32. B. R. Munn, E. Müller, I. Favre–Bulle, E. Scott, J. T. Lizier, M. Breakspear, and J. M. Shine, Multiscale organization of neuronal activity unifies scale-dependent theories of brain function, Cell 187, 7303 (2024).
  33. E. Ganmor, R. Segev, and E. Schneidman, Sparse low-order interaction network underlies a highly correlated and learnable neural population code, Proc. Natl. Acad. Sci. USA 108, 9679 (2011).
  34. J. Humplik and G. Tkačik, Probabilistic models for neural populations that naturally capture global coupling and criticality, PLoS Comput. Biol. 13, e1005763 (2017).
  35. O. Maoz, G. Tkačik, M. S. Esteki, R. Kiani, and E. Schneidman, Learning probabilistic neural representations with randomly connected circuits, Proc. Natl. Acad. Sci. USA 117, 25066 (2020).
  36. L. Meshulam and W. Bialek, Statistical mechanics for networks of real neurons, Rev. Mod. Phys. 97, 045002 (2025).
  37. L. G. Valiant, A theory of the learnable, Commun. ACM 27, 1134 (1984).
  38. T. L. H. Watkin, A. Rau, and M. Biehl, The statistical mechanics of learning a rule, Rev. Mod. Phys. 65, 499 (1993).
  39. S. Shalev-Shwartz and S. Ben-David, Understanding Machine Learning: From Theory to Algorithms (Cambridge University Press, Cambridge, UK, 2014).
  40. W. Bialek, S. E. Palmer, and D. J. Schwab, What makes it possible to learn probability distributions in the natural world? arXiv:2008.12279.
  41. L. Ramirez and W. Bialek, Compression as a path to simplification: Models of collective neural activity, arXiv:2112.14334.
  42. A. Grattafiori et al., The Llama 3 herd of models, arXiv:2407.21783.
  43. C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, Understanding deep learning (still) requires rethinking generalization, Commun. ACM 64, 107 (2021).
  44. Y. Zhang, P. Tiňo, A. Leonardis, and K. Tang, Survey on neural network interpretability, IEEE Trans. Emerging Top. Comput. Intell. 5, 726 (2021).
  45. C. E. Shannon, Prediction and entropy of written English, Bell Sys. Tech. J. 30, 50 (1951).
  46. C. E. Shannon, A mathematical theory of communication, Bell Sys. Tech. J. 27, 379 (1948).
  47. T. M. Cover and J. A. Thomas, Elements of Information Theory (Wiley, New York, NY, 1991).
  48. J. Eisert, M. Cramer, and M. B. Plenio, Colloquium: Area laws for the entanglement entropy, Rev. Mod. Phys. 82, 277 (2010).
  49. D. N. Page, Average entropy of a subsystem, Phys. Rev. Lett. 71, 1291 (1993).
  50. B. Derrida, Random-energy model: An exactly solvable model of disordered systems, Phys. Rev. B 24, 2613 (1981).
  51. V. Ngampruetikorn and D. J. Schwab, Random-energy secret sharing via extreme synergy, Phys. Rev. Lett. 133, 257401 (2024).
  52. https://cmd.rcc.uchicago.edu.
  53. N. Tishby, F. C. Pereira, and W. Bialek, The information bottleneck method, in Proceedings of the 37th Annual Allerton Conference on Communication, Control and Computing, edited by B. Hajek and R. S. Sreenivas (University of Illinois, Urbana, IL, 1999), Vol. 37, pp. 368–377.
  54. G. Tkačik, T. Mora, O. Marre, D. Amodei, S. E. Palmer, M. J. Berry II, and W. Bialek, Thermodynamics and signatures of criticality in a network of neurons, Proc. Natl. Acad. Sci. USA 112, 11508 (2015).
  55. D. J. Strouse and D. J. Schwab, The deterministic information bottleneck, Neural Comput. 29, 1611 (2017).
  56. L. P. Kadanoff, Scaling laws for Ising models near Tc, Physics Physique Fizika 2, 263 (1966).
  57. J. Cardy, Scaling and Renormalization in Statistical Physics (Cambridge University Press, Cambridge, UK, 1996).
  58. S. P. Strong, R. Koberle, R. R. de Ruyter van Steveninck, and W. Bialek, Entropy and information in neural spike trains, Phys. Rev. Lett. 80, 197 (1998).
  59. N. Brenner, S. P. Strong, R. Koberle, W. Bialek, and R. R. de Ruyter van Steveninck, Synergy in a neural code, Neural Comput. 12, 1531 (2000).
  60. S. Nirenberg, S. M. Carcieri, A. L. Jacobs, and P. E. Latham, Retinal gan glion cells act largely as independent encoders, Nature (London) 411, 698 (2001).
  61. H. G. Eyherabide and I. Samengo, When and why noise correlations are important in neural decoding, J. Neurosci. 33, 17921 (2013).
  62. H. Sompolinsky, H. Yoon, K. Kang, and M. Shamir, Population coding in neuronal systems with correlated noise, Phys. Rev. E 64, 051904 (2001).
  63. A. Ecker, P. Berens, A. Tolias, and M. Bethge, The effect of noise correlations in populations of diversely tuned neurons, Journ. Neuro. 31, 14272 (2011).
  64. B. B. Averbeck, P. E. Latham, and A. Pouget, Neural correlations, population coding and computation, Nat. Rev. Neurosci. 7, 358 (2006).
  65. I. Kanitscheider, R. Coen-Cagli, and A. Pouget, Origin of information-limiting noise correlations, Proc. Natl. Acad. Sci. USA 112, E6973 (2015).
  66. Y. Hu, J. Zylberberg, and E. Shea-Brown, The sign rule and beyond: Boundary effects, flexibility, and noise correlations in neural population codes, PLoS Comput. Biol. 10, e1003469 (2014).
  67. J. Cafaro and F. Rieke, Noise correlations improve response fidelity and stimulus encoding, Nature (London) 468, 964 (2010).
  68. J. Zylberberg, J. Cafaro, M. H. Turner, E. Shea-Brown, and F. Rieke, Direction-selective circuits shape noise to ensure a precise population code, Neuron 89, 369 (2016).
  69. B. D. Hoshal, C. M. Holmes, K. Bojanek, J. Salisbury, M. J. Berry, O. Marre, and S. E. Palmer, Stimulus-invariant aspects of the retinal code drive discriminability of natural scenes, Proc. Nat. Acad. Sci. 121, e2313676121 (2024).
  70. J. S. Prentice, O. Marre, M. L. Ioffe, A. R. Loback, G. Tkačik, and M. J. Berry, Error-robust modes of the retinal population code, PLoS Comput. Biol. 12, e1005148 (2016).
  71. X. Ding, D. Lee, J. B. Melander, S. Ganguli, and S. A. Baccus, Adaptation of retinal discriminability to natural scenes, bioRxiv (2024).
  72. M. Manu, L. T. McIntosh, D. B. Kastner, B. N. Naecker, and S. A. Baccus, Synchronous inhibitory pathways create both efficiency and diversity in the retina, Proc. Natl. Acad. Sci. USA 119, e2116589119 (2022).
  73. W. Bialek, I. Nemenman, and N. Tishby, Predictability, complexity, and learning, Neural Comput. 13, 2409 (2001).
  74. J. M. Salisbury and S. E. Palmer, A dynamic scale-mixture model of motion in natural scenes, eLife 14, e104054 (2025).
  75. S. E. Palmer, O. Marre, M. J. Berry, and W. Bialek, Predictive information in a sensory population, Proc. Natl. Acad. Sci. USA 112, 6908 (2015).
  76. A. J. Sederberg, J. N. MacLean, and S. E. Palmer, Learning to make external sensory stimulus predictions using internal correlations in populations of neurons, Proc. Natl. Acad. Sci. USA 115, 1105 (2018).
  77. M. Chalk, O. Marre, and G. Tkačik, Toward a unified theory of efficient, predictive, and sparse coding, Proc. Natl. Acad. Sci. USA 115, 186 (2018).
  78. B. Liu, A. Hong, F. Rieke, and M. B. Manookin, Predictive encoding of motion begins in the primate retina, Nat. Neurosci. 24, 1280 (2021).
  79. GitHub repository, https://github.com/luframirezoc/Ramirez_et.al._2026_probabilistic.
  80. W. Bialek, Biophyics: Searching for Principles (Princeton University Press, Princeton, NJ, 2012).

Outline

Information

PRX Life 4, 023008– Published 23 April, 2026
  • Received 11 April 2022
  • Accepted 24 March 2026
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation