- Physical Review Journals
All JournalsPhysics Magazine

PRX Life

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Hidden Data Structures and Statistics of Natural Images

Philipp Fleig*

Ilya Nemenman

  • Department of Physics, Emory University, Atlanta, Georgia 30322, USA; Department of Biology, Emory University, Atlanta, Georgia 30322, USA; and Initiative in Theory and Modeling of Living Systems, Atlanta, Georgia 30322, USA

  • *Contact author: philipp.fleig@mr.mpg.de
PDF

PRX Life 3, 023014 – Published 12 June, 2025

DOI: https://doi.org/10.1103/8xd4-6hsr

Abstract

Complex, high-dimensional data are often analyzed by grouping their constituent units into components, sometimes referred to as latent features, which afford physical or biological interpretation. However, a priori many different types of latent features and data decompositions can be defined, and one typically uses a trial-and-error approach to determine a decomposition that is natural to the system and its data. A principled understanding of which decomposition is appropriate for a given dataset is needed. In this work, we take a step in this direction and argue that sample-sample correlations carry important information to this effect. We construct a generative random latent feature matrix model of high-dimensional data based on linear mixing of latent features. As a key ingredient, we allow for a statistical dependence between the mixing coefficients. We argue that, with only two parameters (latent dimensionality and the structure of the statistical dependence), the resulting model captures characteristic properties found in many types of natural data. Specifically, the model can produce (overlapping) clusters, sparse mixing, and constrained (non-negative) mixing. We describe typical correlations and eigenvalue distributions of each pattern. Finally, we fit the model on correlation patterns in a benchmark natural images dataset, and we find a good match between the data and the sparse mixing regime of our model. This is in line with the well-known sparse coding structure in natural images, arguing that our approach identifies a correct latent decomposition of these data. We believe that our work will deliver similar insights for other diverse datasets.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (55)

  1. P. W. Anderson, More is different: Broken symmetry and the nature of the hierarchical structure of science, Science 177, 393 (1972).
  2. O. Marre, D. Amodei, N. Deshmukh, K. Sadeghi, F. Soo, T. E. Holy, and M. J. Berry, II, Mapping a complete neural population in the retina, J. Neurosci. 32, 14859 (2012).
  3. N. A. Steinmetz, C. Aydin, A. Lebedeva, M. Okun, M. Pachitariu, M. Bauza, M. Beau, J. Bhagat, C. Böhm, M. Broux, S. Chen, J. Colonell, R. J. Gardner, B. Karsh, F. Kloosterman, D. Kostadinov, C. Mora-Lopez, J. O'Callaghan, J. Park, J. Putzeys et al., Neuropixels 2.0: A miniaturized high-density probe for stable, long-term brain recordings, Science 372, eabf4588 (2021).
  4. M. Nagy, H. Naik, F. Kano, N. V. Carlson, J. C. Koblitz, M. Wikelski, and I. D. Couzin, Smart-barn: Scalable multimodal arena for real-time tracking behavior of animals in large numbers, Sci. Adv. 9, eadf8068 (2023).
  5. A. Cavagna, X. Feng, S. Melillo, L. Parisi, L. Postiglione, and P. Villegas, Como: A novel comoving 3d camera system, IEEE Trans. Instrum. Meas. 70, 1 (2021).
  6. E. Schneidman, M. J. Berry, II, R. Segev, and W. Bialek, Weak pairwise correlations imply strongly correlated network states in a neural population, Nature (London) 440, 1007 (2006).
  7. L. Meshulam, J. L. Gauthier, C. D. Brody, D. W. Tank, and W. Bialek, Coarse graining, fixed points, and scaling in a large population of neurons, Phys. Rev. Lett. 123, 178103 (2019).
  8. D. L. Ruderman, Origins of scaling in natural images, Vis. Res. 37, 3385 (1997).
  9. N. Halabi, O. Rivoire, S. Leibler, and R. Ranganathan, Protein sectors: Evolutionary units of three-dimensional structure, Cell 138, 774 (2009).
  10. D. S. Marks, L. J. Colwell, R. Sheridan, T. A. Hopf, A. Pagnani, R. Zecchina, and C. Sander, Protein 3d structure computed from evolutionary sequence variation, PLoS ONE 6, e28766 (2011).
  11. D. L. Ruderman and W. Bialek, Statistics of natural images: Scaling in the woods, Phys. Rev. Lett. 73, 814 (1994).
  12. C. Qin and L. J. Colwell, Power law tails in phylogenetic systems, Proc. Natl. Acad. Sci. USA 115, 690 (2018).
  13. M. Nitzan and M. P. Brenner, Revealing lineage-related signals in single-cell gene expression using random matrix theory, Proc. Natl. Acad. Sci. USA 118, e1913931118 (2021).
  14. U. Tomasini and M. Wyart, How deep networks learn sparse and hierarchical data: The sparse random hierarchy model, arXiv:2404.10727.
  15. http://www.rctn.org/bruno/sparsenet/.
  16. B. A. Olshausen and D. J. Field, Emergence of simple-cell receptive field properties by learning a sparse code for natural images, Nature (London) 381, 607 (1996).
  17. J. P. Cunningham and B. M. Yu, Dimensionality reduction for large-scale neural recordings, Nat. Neurosci. 17, 1500 (2014).
  18. P. Gao, E. Trautmann, B. Yu, G. Santhanam, S. Ryu, K. Shenoy, and S. Ganguli, A theory of multineuronal dimensionality, dynamics and measurement, bioRxiv (2017), doi: 10.1101/214262.
  19. J. A. Gallego, M. G. Perich, L. E. Miller, and S. A. Solla, Neural manifolds for the control of movement, Neuron 94, 978 (2017).
  20. S. M. Perkins, E. A. Amematsro, J. Cunningham, Q. Wang, and M. M. Churchland, An emerging view of neural geometry in motor cortex supports high-performance decoding, eLife 12, RP89421 (2025).
  21. C. Pandarinath, D. J. O'Shea, J. Collins, R. Jozefowicz, S. D. Stavisky, J. C. Kao, E. M. Trautmann, M. T. Kaufman, S. I. Ryu, L. R. Hochberg et al., Inferring single-trial neural population dynamics using sequential auto-encoders, Nat. Methods 15, 805 (2018).
  22. M. C. Morrell, A. J. Sederberg, and I. Nemenman, Latent dynamical variables produce signatures of spatiotemporal criticality in large biological systems, Phys. Rev. Lett. 126, 118302 (2021).
  23. E. H. Nieh, M. Schottdorf, N. W. Freeman, R. J. Low, S. Lewallen, S. A. Koay, L. Pinto, J. L. Gauthier, C. D. Brody, and D. W. Tank, Geometry of abstract learned knowledge in the hippocampus, Nature (London) 595, 80 (2021).
  24. G. J. Stephens, B. Johnson-Kerner, W. Bialek, and W. S. Ryu, Dimensionality and dynamics in the behavior of c. elegans, PLoS Comput. Biol. 4, e1000028 (2008).
  25. J. Moran and M. Tikhonov, Defining coarse-grainability in a model of structured microbial ecosystems, Phys. Rev. X 12, 021038 (2022).
  26. J. Moran and M. Tikhonov, Emergent predictability in microbial ecosystems, arXiv:2403.19372.
  27. D. Jordan, S. Kuehn, E. Katifori, and S. Leibler, Behavioral diversity in microbes and low-dimensional phenotypic spaces, Proc. Natl. Acad. Sci. USA 110, 14018 (2013).
  28. V. Ngampruetikorn, I. Nemenman, and D. J. Schwab, Extrinsic vs intrinsic criticality in systems with many components, Phys. Rev. Res. 7, 013188 (2025).
  29. M. B. Eisen, P. T. Spellman, P. O. Brown, and D. Botstein, Cluster analysis and display of genome-wide expression patterns, Proc. Natl. Acad. Sci. USA 95, 14863 (1998).
  30. S. Goldt, M. Mézard, F. Krzakala, and L. Zdeborová, Modeling the influence of data structure on learning in neural networks: The hidden manifold model, Phys. Rev. X 10, 041044 (2020).
  31. D. D. Lee and H. S. Seung, Learning the parts of objects by non-negative matrix factorization, Nature (London) 401, 788 (1999).
  32. M. Potters and J.-P. Bouchaud, A First Course in Random Matrix Theory: For Physicists, Engineers and Data Scientists (Cambridge University Press, Cambridge, 2020).
  33. F. Mignacco, F. Krzakala, Y. Lu, P. Urbani, and L. Zdeborova, The role of regularization in classification of high-dimensional noisy gaussian mixture, in Proceedings of the International Conference on Machine Learning (PMLR, 2020), pp. 6874–6883.
  34. P. Fleig and I. Nemenman, Statistical properties of large data sets with linear latent features, Phys. Rev. E 106, 014102 (2022).
  35. R. B. Grosse, R. Salakhutdinov, W. T. Freeman, and J. B. Tenenbaum, Exploiting compositionality to explore a large space of model structures, in Proceedings of the Twenty-Eighth Conference on Uncertainty in Artificial Intelligence, UAI'12 (AUAI Press, Arlington, VA, 2012), pp. 306–315.
  36. I. D. Landau, G. C. Mel, and S. Ganguli, Singular vectors of sums of rectangular random matrices and optimal estimation of high-rank signals: The extensive spike model, Phys. Rev. E 108, 054129 (2023).
  37. B. A. Olshausen and D. J. Field, Sparse coding with an overcomplete basis set: A strategy employed by v1? Vis. Res. 37, 3311 (1997).
  38. I. Nemenman, F. Shafee, and W. Bialek, Entropy and inference, revisited, in Advances in Neural Information Processing Systems, edited by T. Dietterich, S. Becker, and Z. Ghahramani (MIT Press, Cambridge, MA, 2001), Vol. 14.
  39. D. M. Blei, A. Y. Ng, and M. I. Jordan, Latent dirichlet allocation, J. Mach. Learn. Res. 3, 993 (2003).
  40. N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen et al., Toy models of superposition, arXiv:2209.10652.
  41. https://en.wikipedia.org/wiki/Dirichlet_distribution.
  42. M. J. Berry and G. Tkačik, Clustering of neural activity: A design principle for population codes, Front. Comput. Neurosci. 14, 20 (2020).
  43. https://docs.scipy.org/doc/scipy/reference/generated/scipy.cluster.hierarchy.linkage.html (SciPy v1.7.1).
  44. L. Laloux, P. Cizeau, J.-P. Bouchaud, and M. Potters, Noise dressing of financial correlation matrices, Phys. Rev. Lett. 83, 1467 (1999).
  45. H. Hotelling, New light on the correlation coefficient and its transforms, J. R. Stat. Soc. Ser. B (Methodol.) 15, 193 (1953).
  46. J. H. van Hateren and D. L. Ruderman, Independent component analysis of natural image sequences yields spatio-temporal filters similar to simple cells in primary visual cortex, Proc. R. Soc. London B 265, 2315 (1998).
  47. V. A. Marčenko and L. A. Pastur, Distribution of eigenvalues for some sets of random matrices, Math. USSR-Sborn. 1, 457 (1967).
  48. S. Saremi and T. J. Sejnowski, Hierarchical model of natural images and the origin of scale invariance, Proc. Natl. Acad. Sci. USA 110, 3071 (2013).
  49. W. Bialek, A. Cavagna, I. Giardina, T. Mora, E. Silvestri, M. Viale, and A. M. Walczak, Statistical mechanics for natural flocks of birds, Proc. Natl. Acad. Sci. USA 109, 4786 (2012).
  50. Y. Gao, E. W. Archer, L. Paninski, and J. P. Cunningham, Linear dynamical neural population models through nonlinear embeddings, in Advances in Neural Information Processing Systems, edited by D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Curran Associates, Red Hook, NY, 2016), Vol. 29.
  51. J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, Deep unsupervised learning using nonequilibrium thermodynamics, in Proceedings of the International Conference on Machine Learning (PMLR, 2015), pp. 2256–2265.
  52. A. J. Bell and T. J. Sejnowski, An information-maximization approach to blind separation and blind deconvolution, Neural Comput. 7, 1129 (1995).
  53. A. Hyvärinen and E. Oja, Independent component analysis: Algorithms and applications, Neural Netw. 13, 411 (2000).
  54. J. Shlens, A tutorial on principal component analysis, arXiv:1404.1100.
  55. W. Młynarski and G. Tkačik, Efficient coding theory of dynamic attentional modulation, PLoS Biol. 20, e3001889 (2022).

Outline

Information

PRX Life 3, 023014– Published 12 June, 2025
  • Received 13 June 2024
  • Accepted 14 May 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation