- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Letters

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Editors' Suggestion
  • Open Access

Surveying the Space of Descriptions of a Composite System with Machine Learning

Kieran A. Murphy1, Yujing Zhang2, and Dani S. Bassett1,3,4,5,6,7,8

PDF

Phys. Rev. Lett. 134, 257401 – Published 27 June, 2025

DOI: https://doi.org/10.1103/gxrh-2xsv

Abstract

Multivariate information theory provides a general and principled framework for understanding how the components of a system are connected. Existing analyses are coarse in nature—built up from characterizations of discrete subsystems—and can be computationally prohibitive. In this work, we propose to study the continuous space of possible descriptions of a composite system as a window into its organizational structure. A description consists of specific information conveyed about each of the components, and the space of possible descriptions is equivalent to the space of lossy compression schemes of the components. We introduce a machine-learning framework to optimize descriptions that extremize key information theoretic quantities used to characterize organization, such as total correlation and O-information. Through case studies on spin systems, sudoku boards, and letter sequences from natural language, we identify extremal descriptions that reveal how system-wide variation emerges from individual components. By integrating machine learning into a fine-grained information theoretic analysis of composite random variables, our framework opens a new avenues for probing the structure of real-world complex systems.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (45)

  1. G. Tononi, O. Sporns, and G. M. Edelman, A measure for brain complexity: Relating functional segregation and integration in the nervous system, Proc. Natl. Acad. Sci. U.S.A. 91, 5033 (1994).
  2. G. Nicolis and C. Nicolis, Foundations of Complex Systems: Emergence, Information and Predicition (World Scientific, Singapore, 2012).
  3. J. Ladyman, J. Lambert, and K. Wiesner, What is a complex system?, Eur. J. Philos. Sci. 3, 33 (2013).
  4. B. C. Daniels, C. J. Ellison, D. C. Krakauer, and J. C. Flack, Quantifying collectivity, Curr. Opin. Neurobiol. 37, 106 (2016).
  5. F. E. Rosas, P. A. M. Mediano, M. Gastpar, and H. J. Jensen, Quantifying high-order interdependencies via multivariate extensions of the mutual information, Phys. Rev. E 100, 032305 (2019).
  6. D. A. Ehrlich, A. C. Schneider, V. Priesemann, M. Wibral, and A. Makkeh, A measure of the complexity of neural representations based on partial information decomposition, Trans. Mach. Learn. Res. (2023), https://openreview.net/forum?id=R8TU3pfzFr.
  7. L. Martignon, G. Deco, K. Laskey, M. Diamond, W. Freiwald, and E. Vaadia, Neural coding: Higher-order temporal patterns in the neurostatistics of cell assemblies, Neural Comput. 12, 2621 (2000).
  8. M. Wibral, C. Finn, P. Wollstadt, J. T. Lizier, and V. Priesemann, Quantifying information modification in developing neural networks via partial information decomposition, Entropy 19, 494 (2017).
  9. T. F. Varley, M. Pope, M. Grazia, Joshua, and O. Sporns, Partial entropy decomposition reveals higher-order information structures in human brain activity, Proc. Natl. Acad. Sci. U.S.A. 120, e2300888120 (2023).
  10. A. I. Luppi, F. E. Rosas, P. A. Mediano, D. K. Menon, and E. A. Stamatakis, Information decomposition and the informational architecture of the brain, Trends Cognit. Sci. 28, 352 (2024).
  11. M. Pope, T. F. Varley, and O. Sporns, Time-varying synergy/redundancy dominance in the human cerebral cortex, J. Phys. 6, 015015 (2025).
  12. J. M. Miller, X. R. Wang, J. T. Lizier, M. Prokopenko, and L. F. Rossi, Measuring information dynamics in swarms, in Guided Self-Organization: Inception (Springer, New York, 2014), pp. 343–364.
  13. K. Pilkiewicz, B. Lemasson, M. Rowland, A. Hein, J. Sun, A. Berdahl, M. Mayo, J. Moehlis, M. Porfiri, E. Fernández-Juricic et al., Decoding collective communications using information theory tools, J. R. Soc. Interface 17, 20190563 (2020).
  14. C. R. Twomey, A. T. Hartnett, M. M. Sosna, and P. Romanczuk, Searching for structure in collective systems, Theor. Biosci. 140, 361 (2021).
  15. A. L. Burns, T. M. Schaerf, J. Lizier, S. Kawaguchi, M. Cox, R. King, J. Krause, and A. J. Ward, Self-organization and information transfer in Antarctic krill swarms, Proc. R. Soc. B 289, 20212361 (2022).
  16. T. E. Chan, M. P. Stumpf, and A. C. Babtie, Gene regulatory network inference from single-cell data using multivariate information measures, Cell syst. 5, 251 (2017).
  17. S. Sootla, D. O. Theis, and R. Vicente, Analyzing information distribution in complex systems, Entropy 19, 636 (2017).
  18. T. Scagliarini, D. Nuzzi, Y. Antonacci, L. Faes, F. E. Rosas, D. Marinazzo, and S. Stramaglia, Gradients of o-information: Low-order descriptors of high-order dependencies, Phys. Rev. Res. 5, 013025 (2023).
  19. S. Watanabe, Information theoretical analysis of multivariate correlation, IBM J. Res. Dev. 4, 66 (1960).
  20. R. G. James, C. J. Ellison, and J. P. Crutchfield, Anatomy of a bit: Information in a time series observation, Chaos 21, 037109 (2011).
  21. R. Marois and J. Ivanoff, Capacity limits of information processing in the brain, Trends Cognit. Sci. 9, 296 (2005).
  22. P. L. Williams and R. D. Beer, Nonnegative decomposition of multivariate information, arXiv:1004.2515.
  23. R. A. Ince, The partial entropy decomposition: Decomposing multivariate entropy and mutual information via pointwise common surprisal, arXiv:1702.01591.
  24. N. Timme, W. Alford, B. Flecker, and J. M. Beggs, Synergy, redundancy, and multivariate information measures: An experimentalist’s perspective, J. Comput. Neurosci. 36, 119 (2014).
  25. A. Kolchinsky, Partial information decomposition: Redundancy as information bottleneck, Entropy 26, 546 (2024).
  26. A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy, Deep variational information bottleneck, in International Conference on Learning Representations (2017), https://openreview.net/forum?id=HyxQzBceg.
  27. M. Koch-Janusz and Z. Ringel, Mutual information, neural networks and the renormalization group, Nat. Phys. 14, 578 (2018).
  28. K. A. Murphy and D. S. Bassett, Interpretability with full complexity by constraining feature information, in International Conference on Learning Representations (ICLR) (2023), https://openreview.net/forum?id=R_OL5mLhsv.
  29. K. A. Murphy and D. S. Bassett, Information decomposition in complex systems via machine learning, Proc. Natl. Acad. Sci. U.S.A. 121, e2312988121 (2024).
  30. K. A. Murphy and D. S. Bassett, Machine-learning optimized measurements of chaotic dynamical systems via the information bottleneck, Phys. Rev. Lett. 132, 197201 (2024).
  31. T. M. Cover and J. A. Thomas, Elements of Information Theory (John Wiley & Sons, New York, 1999).
  32. C. P. Burgess, I. Higgins, A. Pal, L. Matthey, N. Watters, G. Desjardins, and A. Lerchner, Understanding disentangling in β-VAE, arXiv:1804.03599.
  33. B. Poole, S. Ozair, A. Van Den Oord, A. Alemi, and G. Tucker, On variational bounds of mutual information, in International Conference on Machine Learning (PMLR, 2019), pp. 5171–5180.
  34. A. v. d. Oord, Y. Li, and O. Vinyals, Representation learning with contrastive predictive coding, arXiv:1807.03748.
  35. T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, A simple framework for contrastive learning of visual representations, in International Conference on Machine Learning (PMLR, 2020), pp. 1597–1607.
  36. C. P. Gomes, B. Selman, N. Crato, and H. Kautz, Heavy-tailed phenomena in satisfiability and constraint satisfaction problems, J. Autom. Reasoning 24, 67 (2000).
  37. M. Ercsey-Ravasz and Z. Toroczkai, The chaos within sudoku, Sci. Rep. 2, 725 (2012).
  38. M. Varga, R. Sumi, Z. Toroczkai, and M. Ercsey-Ravasz, Order-to-chaos transition in the hardness of random Boolean satisfiability problems, Phys. Rev. E 93, 052211 (2016).
  39. See Supplemental Material at http://link.aps.org/supplemental/10.1103/gxrh-2xsv for implementation specifics and additional results for the sudoku and 4-gram systems.
  40. T. Kailath, The divergence and Bhattacharyya distance measures in signal selection, IEEE Trans. Commun. Tech. 15, 52 (1967).
  41. K. A. Murphy, S. Dillavou, and D. S. Bassett, Comparing the information content of probabilistic representation spaces, Trans. Mach. Learn. Res. (2025), https://openreview.net/forum?id=adhsMqURI1.
  42. C. E. Shannon, A mathematical theory of communication, Bell Syst. Tech. J. 27, 379 (1948).
  43. G. J. Stephens and W. Bialek, Statistical mechanics of letters in words, Phys. Rev. E 81, 066119 (2010).
  44. T. Scagliarini, D. Marinazzo, Y. Guo, S. Stramaglia, and F. E. Rosas, Quantifying high-order interdependencies on individual patterns via the local o-information: Theory and applications to music analysis, Phys. Rev. Res. 4, 013184 (2022).
  45. S. Nirenberg, S. M. Carcieri, A. L. Jacobs, and P. E. Latham, Retinal ganglion cells act largely as independent encoders, Nature (London) 411, 698 (2001).

Outline

Information

Phys. Rev. Lett. 134, 257401– Published 27 June, 2025
  • Received 7 December 2024
  • Revised 22 April 2025
  • Accepted 29 May 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation