- Physical Review Journals
All JournalsPhysics Magazine

PRX Life

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Maximizing Memory Capacity in Heterogeneous Networks

Kaining Zhang1 and Gaia Tavoni1,2,3,*

  • *Contact author: gaia.tavoni@wustl.edu
PDF

PRX Life 3, 023016 – Published 20 June, 2025

DOI: https://doi.org/10.1103/qb9h-qxkx

Abstract

A central question in neuroscience is which neuronal and connectivity properties determine a network's ability to store information. The theory of Hopfield models has provided powerful tools to investigate this question. Here, we generalize classical results on the memory capacity of homogeneous Hopfield models to a broader class of heterogeneous networks. We derive an analytical formula for the maximal capacity of networks with arbitrary, and generally heterogeneous, activation rates (coding levels) of neurons in memory patterns, as well as arbitrary connectivity architectures. We use this result to make normative predictions about the properties that maximize capacity in brain-inspired networks. We show that, although heterogeneity in neuron coding levels and inward connection counts (in-degrees) generally reduces capacity, maximal capacity is retained when these two parameters are correlated. This prediction holds across various biologically relevant scenarios, including the storage of independent patterns in both classical and dendritic networks, as well as the storage of example patterns clustered around prototypes representing concepts. In the latter case, heterogeneity similarly affects the capacity for both examples and concepts. Finally, we analyze bipartite models of the CA3-DG circuit in the hippocampus, known to be crucial for memory encoding. In these networks, capacity is maximized by a quasi-indexing encoding scheme, where each neuron in the DG binds subsets of features from a few memory patterns stored in CA3. Compared to a complete-indexing scheme, where each DG unit binds all the features of a single pattern, quasi-indexing significantly improves both network capacity and memory robustness to neuron ablation. These findings introduce new dimensions to the hippocampal index theory of memory encoding and retrieval, while suggesting specific neural substrates for these functions.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (105)

  1. J. J. Hopfield, Neural networks and physical systems with emergent collective computational abilities, Proc. Natl. Acad. Sci. USA 79, 2554 (1982).
  2. W. A. Little, The existence of persistent states in the brain, Math. Biosci. 19, 101 (1974).
  3. S. I. Amari, Learning patterns and pattern sequences by self-organizing nets of threshold elements, IEEE Trans. Comput. C-21, 1197 (1972).
  4. S. G. Brush, History of the Lenz-Ising model, Rev. Mod. Phys. 39, 883 (1967).
  5. D. J. Amit, Modeling Brain Function: The World of Attractor Neural Networks (Cambridge University Press, Cambridge, 1989).
  6. C. Marullo and E. Agliari, Boltzmann machines as generalized Hopfield networks: A review of recent results and outlooks, Entropy 23, 34 (2020).
  7. N. Brunel, Is cortical connectivity optimized for storing information?, Nat. Neurosci. 19, 749 (2016).
  8. D. Krotov and J. J. Hopfield, Large associative memory problem in neurobiology and machine learning, arXiv:2008.06996.
  9. D. Krotov, Hierarchical associative memory, arXiv:2107.06446.
  10. D. Krotov and J. J. Hopfield, Dense associative memory for pattern recognition, in Advances in Neural Information Processing Systems, edited by D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Curran Associates, Red Hook, NY, 2016), Vol. 29.
  11. M. Gabrié, S. Ganguli, C. Lucibello, and R. Zecchina, Neural networks: From the perceptron to deep nets, in Spin Glass Theory and Far Beyond: Replica Symmetry Breaking After 40 Years (World Scientific, Singapore, 2023), pp. 477–497.
  12. S. Cocco, R. Monasson, L. Posani, S. Rosay, and J. Tubiana, Statistical physics and representations in real and artificial neural networks, Physica A 504, 45 (2018).
  13. S. Fusi, Memory capacity of neural network models, arXiv:2108.07839.
  14. M. K. Benna and S. Fusi, Place cells may simply be memory cells: Memory compression leads to spatial tuning and history dependence, Proc. Natl. Acad. Sci. USA 118, e2018422118 (2021).
  15. D. O. Hebb, The Organization of Behavior: A Neuropsychological Theory (Psychology Press, London, UK, 2005).
  16. D. J. Amit, H. Gutfreund, and H. Sompolinsky, Storing infinite numbers of patterns in a spin-glass model of neural networks, Phys. Rev. Lett. 55, 1530 (1985).
  17. D. J. Amit, H. Gutfreund, and H. Sompolinsky, Statistical mechanics of neural networks near saturation, Ann. Phys. 173, 30 (1987).
  18. E. Gardner, The space of interactions in neural network models, J. Phys. A 21, 257 (1988).
  19. E. Gardner, Maximum storage capacity in neural networks, Europhys. Lett. 4, 481 (1987).
  20. E. Gardner and B. Derrida, Optimal storage properties of neural network models, J. Phys. A 21, 271 (1988).
  21. H. Sompolinsky, Neural networks with nonlinear synapses and a static noise, Phys. Rev. A 34, 2571 (1986).
  22. B. Wemmenhove and A. C. C. Coolen, Finite connectivity attractor neural networks, J. Phys. A 36, 9617 (2003).
  23. D. J. Amit, M. R. Evans, H. Horner, and K. Y. M. Wong, Retrieval phase diagrams for attractor neural networks with optimal interactions, J. Phys. A 23, 3361 (1990).
  24. Y. Feng and N. Brunel, Storage capacity of networks with discrete synapses and sparsely encoded memories, Phys. Rev. E 105, 054408 (2022).
  25. P. Sollich, D. Tantari, A. Annibale, and A. Barra, Extensive parallel processing on scale-free networks, Phys. Rev. Lett. 113, 238106 (2014).
  26. I. P. Castillo, B. Wemmenhove, J. P. L. Hatchett, A. C. C. Coolen, N. S. Skantzos, and T. Nikoletopoulos, Analytic solution of attractor neural networks on scale-free graphs, J. Phys. A 37, 8789 (2004).
  27. T. M. Cover, Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition, IEEE Trans. Electron. Comput. EC-14, 326 (1965).
  28. P. Del Giudice, S. Franz, and M. A. Virasoro, Perceptron beyond the limit of capacity, J. Phys. 50, 121 (1989).
  29. R. Monasson, Properties of neural networks storing spatially correlated patterns, J. Phys. A 25, 3701 (1992).
  30. N. Brunel, J.-P. Nadal, and G. Toulouse, Information capacity of a perceptron, J. Phys. A 25, 5017 (1992).
  31. S. Y. Chung, D. D. Lee, and H. Sompolinsky, Linear readout of object manifolds, Phys. Rev. E 93, 060301(R) (2016).
  32. S. Y. Chung, D. D. Lee, and H. Sompolinsky, Classification and geometry of general perceptual manifolds, Phys. Rev. X 8, 031003 (2018).
  33. U. Cohen, S. Y. Chung, D. D. Lee, and H. Sompolinsky, Separability and geometry of object manifolds in deep neural networks, Nat. Commun. 11, 746 (2020).
  34. T. Shinzato and Y. Kabashima, Perceptron capacity revisited: classification ability for correlated patterns, J. Phys. A 41, 324013 (2008).
  35. F. Gerace, F. Krzakala, B. Loureiro, L. Stephan, and L. Zdeborová, Gaussian universality of perceptrons with random labels, Phys. Rev. E 109, 034305 (2024).
  36. M. Negri, C. Lauditi, G. Perugini, C. Lucibello, and E. Malatesta, Storage and learning phase transitions in the random-features Hopfield model, Phys. Rev. Lett. 131, 257301 (2023).
  37. S. Kalaj, C. Lauditi, G. Perugini, C. Lucibello, E. M. Malatesta, and M. Negri, Random features Hopfield networks generalize retrieval to previously unseen examples, arXiv:2407.05658.
  38. A. Battista and R. Monasson, Capacity-resolution trade-off in the optimal learning of multiple low-dimensional manifolds by attractor neural networks, Phys. Rev. Lett. 124, 048302 (2020).
  39. B. Lopez, M. Schroder, and M. Opper, Storage of correlated patterns in a perceptron, J. Phys. A 28, L447 (1995).
  40. M. Schröder, W. Kinzel, and I. Kanter, Training a perceptron by a bit sequence: Storage capacity, J. Phys. A 29, 7965 (1996).
  41. L. Kang and T. Toyoizumi, Hopfield-like network with complementary encodings of memories, Phys. Rev. E 108, 054410 (2023).
  42. L. Kang and T. Toyoizumi, Distinguishing examples while building concepts in hippocampal and artificial networks, Nat. Commun. 15, 647 (2024).
  43. E. Barkai, D. Hansel, and H. Sompolinsky, Broken symmetries in multilayered perceptrons, Phys. Rev. A 45, 4146 (1992).
  44. A. Engel, H. M. Köhler, F. Tschepke, H. Vollmayr, and A. Zippelius, Storage capacity and learning algorithms for two-layer neural networks, Phys. Rev. A 45, 7590 (1992).
  45. B. L. Annesi, E. M. Malatesta, and F. Zamponi, Exact full-RSB SAT/UNSAT transition in infinitely wide two-layer neural networks, SciPost Phys. 18, 118 (2025).
  46. C. Baldassi, E. M. Malatesta, and R. Zecchina, Properties of the geometry of solutions and capacity of multilayer neural networks with rectified linear unit activations, Phys. Rev. Lett. 123, 170602 (2019).
  47. J. A. Zavatone-Veth and C. Pehlevan, Activation function dependence of the storage capacity of treelike neural networks, Phys. Rev. E 103, L020301 (2021).
  48. J. A. Zavatone-Veth, Statistical Mechanics of Bayesian Inference and Learning in Neural Networks (Harvard University, Cambridge, MA, 2024).
  49. M. Stojnic, Fixed width treelike neural networks capacity analysis–generic activations, arXiv:2402.05696.
  50. B. Kosko, Bidirectional associative memories, IEEE Trans. Syst. Man. Cybern. 18, 49 (1988).
  51. D. H. Ackley, G. E. Hinton, and T. J. Sejnowski, A learning algorithm for Boltzmann machines, Cogn. Sci. 9, 147 (1985).
  52. G. E. Hinton, A practical guide to training restricted Boltzmann machines, in Neural Networks: Tricks of the Trade, 2nd ed. (Springer, Berlin, 2012), pp. 599–619.
  53. T. D. Goode, K. Z. Tanaka, A. Sahay, and T. J. McHugh, An integrated index: Engrams, place cells, and hippocampal memory, Neuron 107, 805 (2020).
  54. K. Nakazawa, M. C. Quirk, R. A. Chitwood, M. Watanabe, M. F. Yeckel, L. D. Sun, A. Kato, C. A. Carr, D. Johnston, M. A. Wilson et al., Requirement for hippocampal CA3 NMDA receptors in associative memory recall, Science 297, 211 (2002).
  55. X. Liu, S. Ramirez, P. T. Pang, C. B. Puryear, A. Govindarajan, K. Deisseroth, and S. Tonegawa, Optogenetic stimulation of a hippocampal engram activates fear memory recall, Nature (London) 484, 381 (2012).
  56. T. J. Teyler and J. W. Rudy, The hippocampal indexing theory and episodic memory: Updating the index, Hippocampus 17, 1158 (2007).
  57. A. Treves and E. T. Rolls, Computational constraints suggest the need for two distinct input systems to the hippocampal CA3 network, Hippocampus 2, 189 (1992).
  58. H. Nishimori, Statistical Physics of Spin Glasses and Information Processing: An Introduction (Clarendon Press, Oxford, UK, 2001), p. 111.
  59. See Supplemental Material at http://link.aps.org/supplemental/10.1103/qb9h-qxkx for additional figures showing the capacity of networks with finite stability, high coding levels, and dendritic structures; the insensitivity of capacity to neuron out-degrees; the optimal coding levels in two-layer networks of finite size; and the capacity of networks with a reversed neuron ratio between the two layers.
  60. M. V. Tsodyks and M. V. Feigel'man, The enhanced storage capacity in neural networks with low activity level, Europhys. Lett. 6, 101 (1988).
  61. M. Mézard, The space of interactions in neural networks: Gardner's computation with the cavity method, J. Phys. A 22, 2181 (1989).
  62. C. Cortes and V. Vapnik, Support-vector networks, Mach. Learn. 20, 273 (1995).
  63. M. Sorbaro, Q. Liu, M. Bortone, and S. Sheik, Optimizing the energy consumption of spiking neural networks for neuromorphic applications, Front. Neurosci. 14, 662 (2020).
  64. Y. Cao, Y. Chen, and D. Khosla, Spiking deep convolutional neural networks for energy-efficient object recognition, Int. J. Comput. Vision 113, 54 (2015).
  65. V. Mehrpour, T. Meyer, E. P. Simoncelli, and N. C. Rust, Pinpointing the neural signatures of single-exposure visual recognition memory, Proc. Natl. Acad. Sci. USA 118, e2021660118 (2021).
  66. K. Zhang, E. S. Bromberg-Martin, F. Sogukpinar, K. Kocher, and I. E. Monosov, Surprise and recency in novelty detection in the primate brain, Curr. Biol. 32, 2160 (2022).
  67. J.-Z. Xiang and M. W. Brown, Differential neuronal encoding of novelty, familiarity and recency in regions of the anterior temporal lobe, Neuropharmacology 37, 657 (1998).
  68. C. Ranganath and G. Rainer, Neural mechanisms for detecting and remembering novel events, Nat. Rev. Neurosci. 4, 193 (2003).
  69. S. Sardi, R. Vardi, A. Sheinin, A. Goldental, and I. Kanter, New types of experiments reveal that a neuron functions as multiple independent threshold units, Sci. Rep. 7, 18036 (2017).
  70. A. Polsky, B. W. Mel, and J. Schiller, Computational subunits in thin dendrites of pyramidal cells, Nat. Neurosci. 7, 621 (2004).
  71. B. Mel, The clusteron: Toward a simple abstraction for a complex neuron, in Advances in Neural Information Processing Systems, edited by J. Moody, S. Hanson, and R. Lippmann (Morgan-Kaufmann, San Francisco, CA, 1991), Vol. 4.
  72. P. Rajasethupathy, S. Sankaran, J. H. Marshel, C. K. Kim, E. Ferenczi, S. Y. Lee, A. Berndt, C. Ramakrishnan, A. Jaffe, M. Lo, C. Liston, and K. Deisseroth, Projections from neocortex mediate top-down control of memory retrieval, Nature (London) 526, 653 (2015).
  73. B. J. Wiltgen and K. Z. Tanaka, Systems consolidation and the content of memory, Neurobiol. Learn. Mem. 106, 365 (2013).
  74. P. A. Lewis and S. J. Durrant, Overlapping memory replay during sleep builds cognitive schemata, Trends Cogn. Sci. 15, 343 (2011).
  75. T. J. Teyler and P. DiScenna, The hippocampal memory indexing theory, Behav. Neurosci. 100, 147 (1986).
  76. J. W. Rudy, Context representations, context functions, and the parahippocampal–hippocampal system, Learn. Mem. 16, 573 (2009).
  77. L. Nadel and M. Moscovitch, Memory consolidation, retrograde amnesia and the hippocampal complex, Curr. Opin. Neurobiol. 7, 217 (1997).
  78. L. R. Squire and P. Alvarez, Retrograde amnesia and memory consolidation: A neurobiological perspective, Curr. Opin. Neurobiol. 5, 169 (1995).
  79. J. L. McClelland, B. L. McNaughton, and R. C. O'Reilly, Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory, Psychol. Rev. 102, 419 (1995).
  80. A. Treves and E. T. Rolls, Computational analysis of the role of the hippocampus in memory, Hippocampus 4, 374 (1994).
  81. R. P. Kesner, A process analysis of the CA3 subregion of the hippocampus, Front. Cell. Neurosci. 7, 78 (2013).
  82. S. M. Attili, K. Moradi, D. W. Wheeler, and G. A. Ascoli, Quantification of neuron types in the rodent hippocampal formation by data mining and numerical optimization, Eur. J. Neurosci. 55, 1724 (2022).
  83. E. T. Rolls, The storage and recall of memories in the hippocampo-cortical system, Cell Tissue Res. 373, 577 (2018).
  84. H. E. Scharfman, The CA3 “backprojection” to the dentate gyrus, Prog. Brain Res. 163, 627 (2007).
  85. M. W. Jung and B. L. McNaughton, Spatial selectivity of unit activity in the hippocampal granular layer, Hippocampus 3, 165 (1993).
  86. J. B. Aimone, W. Deng, and F. H. Gage, Resolving new memories: A critical look at the dentate gyrus, adult neurogenesis, and pattern separation, Neuron 70, 589 (2011).
  87. K. K. Cowansage, T. Shuman, B. C. Dillingham, A. Chang, P. Golshani, and M. Mayford, Direct reactivation of a coherent neocortical memory of context, Neuron 84, 432 (2014).
  88. A. Engel and C. Broeck, Statistical Mechanics of Learning (Cambridge University Press, Cambridge, 2001).
  89. F. Schönsberg, Y. Roudi, and A. Treves, Efficiency of local learning rules in threshold-linear associative networks, Phys. Rev. Lett. 126, 018301 (2021).
  90. G. Tavoni, U. Ferrari, F. P. Battaglia, S. Cocco, and R. Monasson, Functional coupling networks inferred from prefrontal cortex activity show experience-related effective plasticity, Netw. Neurosci. 1, 275 (2017).
  91. G. Tavoni, S. Cocco, and R. Monasson, Neural assemblies revealed by inferred connectivity-based models of prefrontal cortex recordings, J. Comput. Neurosci. 41, 269 (2016).
  92. S. Cocco, R. Monasson, L. Posani, and G. Tavoni, Functional networks from inverse modeling of neural population activity, Curr. Opin. Syst. Biol. 3, 103 (2017).
  93. O. Miry, J. Li, and L. Chen, The quest for the hippocampal memory engram: From theories to experimental evidence, Front. Behav. Neurosci. 14, 632019 (2021).
  94. T. E. J. Behrens, T. H. Muller, J. C. R. Whittington, S. Mark, A. B. Baram, K. L. Stachenfeld, and Z. Kurth-Nelson, What is a cognitive map? Organizing knowledge for flexible behavior, Neuron 100, 490 (2018).
  95. R. M. Tavares, A. Mendelsohn, Y. Grossman, C. H. Williams, M. Shapiro, Y. Trope, and D. Schiller, A map for social navigation in the human brain, Neuron 87, 231 (2015).
  96. D. Sun, N. E. K. Shaik, R. R. Unnithan, and C. French, Hippocampal cognitive and relational map paradigms explored by multisensory encoding recording with wide-field calcium imaging, iScience 27, 108603 (2023).
  97. M. Diamantaki, M. Frey, P. Berens, P. Preston-Ferrer, and A. Burgalossi, Sparse activity of identified dentate granule cells during spatial exploration, eLife 5, e20252 (2016).
  98. A. Sahay, D. A. Wilson, and R. Hen, Pattern separation: A common function for new neurons in hippocampus and olfactory bulb, Neuron 70, 582 (2011).
  99. S. B. McHugh, V. Lopes-dos Santos, G. P. Gava, K. Hartwich, S. K. E. Tam, D. M. Bannerman, and D. Dupret, Adult-born dentate granule cells promote hippocampal population sparsity, Nat. Neurosci. 25, 1481 (2022).
  100. T. Kitamura, Y. Saitoh, N. Takashima, A. Murayama, Y. Niibori, H. Ageta, M. Sekiguchi, H. Sugiyama, and K. Inokuchi, Adult neurogenesis modulates the hippocampus-dependent period of associative fear memory, Cell 139, 814 (2009).
  101. J. B. Aimone, J. Wiles, and F. H. Gage, Potential role for adult neurogenesis in the encoding of time in new memories, Nat. Neurosci. 9, 723 (2006).
  102. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, Scikit-learn: Machine learning in Python, J. Mach. Learn. Res. 12, 2825 (2011).
  103. P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright et al., Scipy 1.0: Fundamental algorithms for scientific computing in Python, Nat. Meth. 17, 261 (2020).
  104. https://github.com/kalfton/heterogeneous_hopfield_network_capacity.git.
  105. C. Ren and A. R. MacKenzie, Closed-form approximations to the error and complementary error functions and their applications in atmospheric science, Atm. Sci. Lett. 8, 70 (2007).

Outline

Information

PRX Life 3, 023016– Published 20 June, 2025
  • Received 25 September 2024
  • Accepted 22 May 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation