Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Connectivity Structure and Dynamics of Nonlinear Recurrent Neural Networks

David G. Clark1,2,*,‡, Owen Marschall1, Alexander van Meegen3,§, and Ashok Litwin-Kumar1,2,†

  • *Contact author: dgclark@fas.harvard.edu
  • †Contact author: a.litwin-kumar@columbia.edu
  • ‡Present address: Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University, Cambridge, Massachusetts 02138, USA.
  • §Present address: School of Life Sciences and School of Computer and Communication Sciences, École Polytechnique Fédérale de Lausanne, 1015 Lausanne, Switzerland.

Phys. Rev. X 15, 041019 – Published 3 November, 2025

DOI: https://doi.org/10.1103/2jt7-c8cq

Abstract

Studies of the dynamics of nonlinear recurrent neural networks often assume independent and identically distributed couplings, but large-scale connectomics data indicate that biological neural circuits exhibit markedly different connectivity properties. These include rapidly decaying singular-value spectra and structured singular-vector overlaps. Here, we develop a theory to analyze how these forms of structure shape high-dimensional collective activity in nonlinear recurrent neural networks. We first introduce the random-mode model, a random-matrix ensemble related to the singular-value decomposition that enables control over the spectrum and right-left mode overlaps. Then, using a novel path-integral calculation, we derive analytical expressions that reveal how connectivity structure affects features of collective dynamics: the dimension of activity, which quantifies the number of high-variance collective-activity fluctuations, and the temporal correlations that characterize the timescales of these fluctuations. We show that connectivity structure can be invisible in single-neuron activities, while dramatically shaping collective activity. Furthermore, despite the nonlinear, high-dimensional nature of these networks, the dimension of activity depends on just two connectivity parameters—the variance of the couplings and the effective rank of the coupling matrix, which quantifies the number of dominant rank-one connectivity components. We contrast the effects of single-neuron heterogeneity and low-dimensional connectivity, making predictions about how z-scoring data affects the dimension of activity. Finally, we demonstrate the presence of structured overlaps between left and right modes in the Drosophila connectome, incorporate them into the theory, and show how they further shape collective dynamics.

View figure in article

Physics Subject Headings (PhySH)

Popular Summary

Article Text

References (105)

  1. O. Barak, D. Sussillo, R. Romo, M. Tsodyks, and L. F. Abbott, From fixed points to chaos: Three models of delayed discrimination, Prog. Neurobiol. 103, 214 (2013).
  2. V. Mante, D. Sussillo, K. V. Shenoy, and W. T. Newsome, Context-dependent computation by recurrent dynamics in prefrontal cortex, Nature (London) 503, 78 (2013).
  3. D. Sussillo, M. M. Churchland, M. T. Kaufman, and K. V. Shenoy, A neural network that finds a naturalistic solution for the production of muscle activity, Nat. Neurosci. 18, 1025 (2015).
  4. S. Hochreiter and J. Schmidhuber, Long short-term memory, Neural Comput. 9, 1735 (1997).
  5. I. Sutskever, O. Vinyals, and Q. V. Le, Sequence to sequence learning with neural networks, Adv. Neural Inf. Process. Syst. 27, 3104 (2014).
  6. K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, Learning phrase representations using RNN encoder–decoder for statistical machine translation, in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), edited by A. Moschitti, B. Pang, and W. Daelemans (Association for Computational Linguistics, Doha, Qatar, 2014), pp. 1724–1734.
  7. K. Krishnamurthy, T. Can, and D. J. Schwab, Theory of gating in recurrent neural networks, Phys. Rev. X 12, 011011 (2022).
  8. K. Rajan, L. F. Abbott, and H. Sompolinsky, Stimulus-dependent suppression of chaos in recurrent neural networks, Phys. Rev. E 82, 011903 (2010).
  9. R. Engelken, F. Wolf, and L. F. Abbott, Lyapunov spectra of chaotic recurrent neural networks, Phys. Rev. Res. 5, 043044 (2023).
  10. H. Sompolinsky, A. Crisanti, and H.-J. Sommers, Chaos in random neural networks, Phys. Rev. Lett. 61, 259 (1988).
  11. D. Martí, N. Brunel, and S. Ostojic, Correlations between synapses in pairs of neurons slow down dynamics in randomly connected neural networks, Phys. Rev. E 97, 062314 (2018).
  12. D. Sussillo and L. F. Abbott, Generating coherent patterns of activity from chaotic neural networks, Neuron 63, 544 (2009).
  13. R. Laje and D. V. Buonomano, Robust timing and motor patterns by taming chaos in recurrent neural networks, Nat. Neurosci. 16, 925 (2013).
  14. J. Aljadeff, M. Stern, and T. Sharpee, Transition to chaos in random networks with cell-type-specific connectivity, Phys. Rev. Lett. 114, 088101 (2015).
  15. B. DePasquale, C. J. Cueva, K. Rajan, G. S. Escola, and L. F. Abbott, full-force: A target-based method for training recurrent networks, PLoS One 13, e0191527 (2018).
  16. T. Asabuki and C. Clopath, Taming the chaos gently: A predictive alignment learning rule in recurrent neural networks, Nat. Commun. 16, 6784 (2025).
  17. L. K. Scheffer, C. S. Xu, M. Januszewski, Z. Lu, S.-y. Takemura, K. J. Hayworth, G. B. Huang, K. Shinomiya, J. Maitlin-Shepard, S. Berg et al., A connectome and analysis of the adult drosophila central brain, eLife 9, e57443 (2020).
  18. S. Loomba, J. Straehle, V. Gangadharan, N. Heike, A. Khalifa, A. Motta, N. Ju, M. Sievers, J. Gempt, H. S. Meyer et al., Connectomic comparison of mouse and human cortex, Science 377, eabo0924 (2022).
  19. A. Shapson-Coe, M. Januszewski, D. R. Berger, A. Pope, Y. Wu, T. Blakely, R. L. Schalek, P. H. Li, S. Wang, J. Maitin-Shepard et al., A petavoxel fragment of human cerebral cortex reconstructed at nanoscale resolution, Science 384, eadk4858 (2024).
  20. The MICrONS Consortium, Functional connectomics spanning multiple areas of mouse visual cortex, Nature (London) 640, 435 (2025).
  21. M. R. Tavakoli, J. Lyudchik, M. Januszewski, V. Vistunou, N. Agudelo Dueñas, J. Vorlaufer, C. Sommer, C. Kreuzinger, B. Oliveira, A. Cenameri et al., Light-microscopy-based connectomic reconstruction of mammalian brain tissue, Nature (London) 642, 398 (2025).
  22. S. Song, P. J. Sjöström, M. Reigl, S. Nelson, and D. B. Chklovskii, Highly nonrandom features of synaptic connectivity in local cortical circuits, PLOS Biol. 3, e68 (2005).
  23. V. Thibeault, A. Allard, and P. Desrosiers, The low-rank hypothesis of complex systems, Nat. Phys. 20, 294 (2024).
  24. I. Akjouj, M. Barbier, M. Clenet, W. Hachem, M. Maïda, F. Massol, J. Najim, and V. C. Tran, Complex systems in ecology: A guided tour with large Lotka–Volterra models and random matrices, Proc. R. Soc. A 480, 20230284 (2024).
  25. Z. Wang, W. Mai, Y. Chai, K. Qi, H. Ren, C. Shen, S. Zhang, G. Tan, Y. Hu, and Q. Wen, The geometry and dimensionality of brain-wide activity, eLife 14, RP100666 (2025).
  26. L. Pezon, V. Schmutz, and W. Gerstner, Linking neural manifolds to circuit structure in recurrent networks, bioRxiv 2024 (2024).
  27. F. Schuessler, F. Mastrogiuseppe, A. Dubreuil, S. Ostojic, and O. Barak, The interplay between randomness and structure during learning in RNNs, Adv. Neural Inf. Process. Syst. 33, 13352 (2020).
  28. C. H. Martin and M. W. Mahoney, Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning, J. Mach. Learn. Res. 22, 1 (2021).
  29. D. G. Clark, L. F. Abbott, and H. Sompolinsky, Symmetries and continuous attractors in disordered neural circuits, bioRxiv 2025 (2025).
  30. E. M. Trautmann, J. K. Hesse, G. M. Stine, R. Xia, S. Zhu, D. J. O’Shea, B. Karsh, J. Colonell, F. F. Lanfranchi, S. Vyas et al., Large-scale high-density brain-wide neural recording in nonhuman primates, Nat. Neurosci. 28, 1562 (2025).
  31. C. Stringer, M. Pachitariu, N. Steinmetz, M. Carandini, and K. D. Harris, High-dimensional geometry of population responses in visual cortex, Nature (London) 571, 361 (2019).
  32. C. Stringer, M. Pachitariu, N. Steinmetz, C. B. Reddy, M. Carandini, and K. D. Harris, Spontaneous behaviors drive multidimensional, brainwide activity, Science 364, eaav7893 (2019).
  33. J. Manley, S. Lu, K. Barber, J. Demas, H. Kim, D. Meyer, F. M. Traub, and A. Vaziri, Simultaneous, cortex-wide dynamics of up to 1 million neurons reveal unbounded scaling of dimensionality with neuron number, Neuron 112, 1694 (2024).
  34. S. Chung and L. F. Abbott, Neural population geometry: An approach for understanding biological and artificial neural networks, Curr. Opin. Neurobiol. 70, 137 (2021).
  35. J. P. Cunningham and M. Y. Byron, Dimensionality reduction for large-scale neural recordings, Nat. Neurosci. 17, 1500 (2014).
  36. P. Gao and S. Ganguli, On simplicity and complexity in the brave new world of large-scale neuroscience, Curr. Opin. Neurobiol. 32, 148 (2015).
  37. E. M. Trautmann, S. D. Stavisky, S. Lahiri, K. C. Ames, M. T. Kaufman, D. J. O’Shea, S. Vyas, X. Sun, S. I. Ryu, S. Ganguli et al., Accurate estimation of neural population dynamics without spike sorting, Neuron 103, 292 (2019).
  38. L. Meshulam and W. Bialek, Statistical mechanics for networks of real neurons, Rev. Mod. Phys. (to be published), 10.1103/jcrn-3nrc.
  39. D. G. Clark, L. F. Abbott, and A. Litwin-Kumar, Dimension of activity in random neural networks, Phys. Rev. Lett. 131, 118401 (2023).
  40. This random structure, rather than exact, deterministic orthonormality, offers analytical tractability by eliminating the need to enforce orthonormality—a complicated global constraint; but see also Ref. [41], which handles random orthonormal matrices in a regression setting.

  41. A. Ingrosso, Optimal learning with excitatory and inhibitory synapses, PLoS Comput. Biol. 16, e1008536 (2020).
  42. F. Mastrogiuseppe and S. Ostojic, Linking connectivity, dynamics, and computations in low-rank recurrent neural networks, Neuron 99, 609 (2018).
  43. F. Schuessler, A. Dubreuil, F. Mastrogiuseppe, S. Ostojic, and O. Barak, Dynamics of random recurrent networks with correlated low-rank structure, Phys. Rev. Res. 2, 013111 (2020).
  44. M. Beiran, A. Dubreuil, A. Valente, F. Mastrogiuseppe, and S. Ostojic, Shaping dynamics with multiple populations in low-rank recurrent networks, Neural Comput. 33, 1572 (2021).
  45. A. Dubreuil, A. Valente, M. Beiran, F. Mastrogiuseppe, and S. Ostojic, The role of population structure in computations through neural dynamics, Nat. Neurosci. 25, 783 (2022).
  46. The Gaussian assumption for individual components is not crucial—our results depend on only first and second moments for large N due to the central limit theorem; higher-order cumulants (scaling appropriately with 1/N) are subdominant in the path-integral saddle-point approximation.

  47. R. Vershynin, High-Dimensional Probability: An Introduction with Applications in Data Science (Cambridge University Press, Cambridge, England, 2018), Vol. 47.
  48. This follows from using Wick’s theorem to evaluate the expectations of tr(LDRTRDLT) and tr(LDRTRDLTLDRTRDLT) for the numerator and denominator, respectively, of PRS.

  49. J. Pennington, S. Schoenholz, and S. Ganguli, Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice, in Advances in Neural Information Processing Systems 30 (NIPS 2017), Long Beach, CA, USA (2017), https://papers.nips.cc/paper_files/paper/2017/hash/d9fc0cdb67638d50f411432d0d41d0ba-Abstract.html.
  50. J. Pennington, S. Schoenholz, and S. Ganguli, The emergence of spectral universality in deep networks, in International Conference on Artificial Intelligence and Statistics (PMLR, 2018), pp. 1924–1932.
  51. This behaves similarly to the more conventional ϕ(x)=tanh(x) but allows for analytical evaluation of Gaussian integrals.

  52. This duality is reminiscent of ergodicity theorems, which relate temporal and spatial averages, but differs in a variety of ways, including that it holds even for finite-time intervals, whereas ergodicity concerns infinite-time limits.

  53. K. Rajan, L. F. Abbott, and H. Sompolinsky, Inferring stimulus selectivity from the spatial structure of neural network dynamics, Adv. Neural Inf. Process. Syst. 23 (2010).
  54. L. F. Abbott, K. Rajan, and H. Sompolinsky, Interactions between intrinsic and stimulus-evoked activity in recurrent neural networks, in The Dynamic Brain: An Exploration of Neuronal Variability and its Functional Significance, edited by M. Ding and D. Glanzman (Oxford University Press, New York, 2011), pp. 65–82.
  55. P. Gao, E. Trautmann, B. Yu, G. Santhanam, S. Ryu, K. Shenoy, and S. Ganguli, A theory of multineuronal dimensionality, dynamics and measurement, bioRxiv 214262 (2017).
  56. A. Litwin-Kumar, K. D. Harris, R. Axel, H. Sompolinsky, and L. F. Abbott, Optimal degrees of synaptic connectivity, Neuron 93, 1153 (2017).
  57. S. Recanatesi, G. K. Ocker, M. A. Buice, and E. Shea-Brown, Dimensionality in recurrent spiking networks: Global trends in activity and local origins in connectivity, PLoS Comput. Biol. 15, 1 (2019).
  58. S. Recanatesi, S. Bradde, V. Balasubramanian, N. A. Steinmetz, and E. Shea-Brown, A scale-dependent measure of system dimensionality, Patterns 3, 100555 (2022).
  59. Y. Hu and H. Sompolinsky, The spectrum of covariance matrices of randomly connected recurrent neuronal networks with linear dynamics, PLoS Comput. Biol. 18, 1 (2022).
  60. A. Crisanti and H. Sompolinsky, Path integral approach to random neural networks, Phys. Rev. E 98, 062120 (2018).
  61. M. Helias and D. Dahmen, Statistical Field Theory for Neural Networks, Lecture Notes in Physics (Springer, Cham, 2020), 10.1007/978-3-030-46444-8.
  62. W. Zou and H. Huang, Introduction to dynamical mean-field theory of randomly connected neural networks with bidirectionally correlated couplings, SciPost Phys. Lect. Notes 079 (2024).
  63. P. C. Martin, E. D. Siggia, and H. A. Rose, Statistical dynamics of classical systems, Phys. Rev. A 8, 423 (1973).
  64. C. De Dominicis, Dynamics as a substitute for replicas in systems with quenched random impurities, Phys. Rev. B 18, 4913 (1978).
  65. J. A. Hertz, Y. Roudi, and P. Sollich, Path integral methods for the dynamics of stochastic and disordered systems, J. Phys. A 50, 033001 (2016).
  66. T. Hromádka, M. R. DeWeese, and A. M. Zador, Sparse representation of sounds in the unanesthetized auditory cortex, PLOS Biol. 6, e16 (2008).
  67. This symmetry arises because the dynamics of the network are determined by an effective coupling matrix LDRTG, where D=diag(D1,…,DM) and G=diag(G1,…,GN). Swapping the distributions of Da and Gi does not change the spectrum when L and R are i.i.d., since LDRTG and RTGLD have the same nonzero eigenvalues.

  68. B. Rider and C. D. Sinclair, Extremal laws for the real ginibre ensemble, Ann. Appl. Probab. 24, 1621 (2014).
  69. D. G. Clark, Transient dynamics of associative memory models, arXiv:2506.05303.
  70. If model neurons represent clusters of biological neurons, strong self-connections may instead be interpreted as within-cluster coupling [71, 72].

  71. M. Stern, H. Sompolinsky, and L. F. Abbott, Dynamics of random neural networks with bistable units, Phys. Rev. E 90, 062710 (2014).
  72. A. Litwin-Kumar and B. Doiron, Slow dynamics and high variability in balanced cortical networks with clustered connections, Nat. Neurosci. 15, 1498 (2012).
  73. R. Milo, S. Shen-Orr, S. Itzkovitz, N. Kashtan, D. Chklovskii, and U. Alon, Network motifs: simple building blocks of complex networks, Science 298, 824 (2002).
  74. Y. Hu, J. Trousdale, K. Josić, and E. Shea-Brown, Motif statistics and spike correlations in neuronal networks, J. Stat. Mech. (2013) P03012.
  75. P. V. Aceituno, T. Rogers, and H. Schomerus, Universal hypotrochoidic law for random matrices with cyclic correlations, Phys. Rev. E 100, 010302(R) (2019).
  76. E. S. Schaffer, N. Mishra, M. R. Whiteway, W. Li, M. B. Vancura, J. Freedman, K. B. Patel, V. Voleti, L. Paninski, E. M. Hillman et al., The spatial and temporal structure of neural activity across the fly brain, Nat. Commun. 14, 5572 (2023).
  77. E. Schneidman, M. J. Berry, R. Segev, and W. Bialek, Weak pairwise correlations imply strongly correlated network states in a neural population, Nature (London) 440, 1007 (2006).
  78. O. Maoz, G. Tkačik, M. S. Esteki, R. Kiani, and E. Schneidman, Learning probabilistic neural representations with randomly connected circuits, Proc. Natl. Acad. Sci. U.S.A. 117, 25066 (2020).
  79. E. Gardner, The space of interactions in neural network models, J. Phys. A 21, 257 (1988).
  80. B. Bordelon, J. Zavatone-Veth, and C. Pehlevan, Mean field theory of representation learning in large rnns, in Cosyne Abstracts 2024 (2024) abstract 3-038.
  81. G. R. Yang, M. R. Joglekar, H. F. Song, W. T. Newsome, and X.-J. Wang, Task representations in neural networks trained to perform many cognitive tasks, Nat. Neurosci. 22, 297 (2019).
  82. L. N. Driscoll, K. Shenoy, and D. Sussillo, Flexible multitask computation in recurrent networks utilizes shared dynamical motifs, Nat. Neurosci. 27, 1349 (2024).
  83. E. Marder and J.-M. Goaillard, Variability, compensation and homeostasis in neuron and network function, Nat. Rev. Neurosci. 7, 563 (2006).
  84. C. Teeter, R. Iyer, V. Menon, N. Gouwens, D. Feng, J. Berg, A. Szafer, N. Cain, H. Zeng, M. Hawrylycz et al., Generalized leaky integrate-and-fire models classify multiple neuron types, Nat. Commun. 9, 709 (2018).
  85. B. K. Murphy and K. D. Miller, Balanced amplification: A new mechanism of selective amplification of neural activity patterns, Neuron 61, 635 (2009).
  86. G. Hennequin, T. P. Vogels, and W. Gerstner, Optimal control of transient dynamics in balanced networks supports generation of complex movements, Neuron 82, 1394 (2014).
  87. K. Rajan and L. F. Abbott, Eigenvalue spectra of random matrices for neural networks, Phys. Rev. Lett. 97, 188104 (2006).
  88. J. Kadmon and H. Sompolinsky, Transition to chaos in random neuronal networks, Phys. Rev. X 5, 041030 (2015).
  89. L. Tiberi, D. Dahmen, and M. Helias, Hidden connectivity structures control collective network dynamics, arXiv:2303.02476.
  90. V. Schmutz, J. Brea, and W. Gerstner, Emergent rate-based dynamics in duplicate-free populations of spiking neurons, Phys. Rev. Lett. 134, 018401 (2025).
  91. Z. Lin and H. Huang, Spiking mode-based neural networks, Phys. Rev. E 110, 024306 (2024).
  92. N. Eckstein, A. S. Bates, A. Champion, M. Du, Y. Yin, P. Schlegel, A. K.-Y. Lu, T. Rymer, S. Finley-May, T. Paterson et al., Neurotransmitter classification from electron microscopy images at synaptic sites in drosophila melanogaster, Cell 187, 2574 (2024).
  93. W. W. Liu and R. I. Wilson, Glutamate is an inhibitory neurotransmitter in the drosophila olfactory system, Proc. Natl. Acad. Sci. U.S.A. 110, 10294 (2013).
  94. S. Molina-Obando, J. F. Vargas-Fique, M. Henning, B. Gür, T. M. Schladt, J. Akhtar, T. K. Berger, and M. Silies, On selectivity in the drosophila visual system is a multisynaptic process involving both glutamatergic and gabaergic inhibition, eLife 8, e49373 (2019).
  95. O. Roy and M. Vetterli, The effective rank: A measure of effective dimensionality, in 2007 15th European Signal Processing Conference, Poznan, Poland (IEEE, 2007), pp. 606–610.
  96. G. Christodoulou, T. P. Vogels, and E. J. Agnes, Regimes and mechanisms of transient amplification in abstract and biological neural networks, PLoS Comput. Biol. 18, e1010365 (2022).
  97. M. Mézard, G. Parisi, and M. A. Virasoro, Spin Glass Theory and Beyond: An Introduction to the Replica Method and Its Applications (World Scientific Publishing Company, Singapore, 1987), Vol. 9.
  98. R. Price, A useful theorem for nonlinear devices having Gaussian inputs, IRE Trans. Inf. Theory 4, 69 (1958).
  99. A. van Meegen, T. Kühn, and M. Helias, Large-deviation approach to random recurrent neuronal networks: Parameter inference and fluctuation-induced transitions, Phys. Rev. Lett. 127, 158302 (2021).
  100. M. A. Buice and J. D. Cowan, Field-theoretic approach to fluctuation effects in neural networks, Phys. Rev. E 75, 051919 (2007).
  101. Note that the variance of ηz(t) is scaled by D2; equivalently, we could factor out D2 from the variance and include D as a multiplicative factor in the latent-variable single-site dynamics [Eq. (i17)].

  102. J. P. Stroud, M. A. Porter, G. Hennequin, and T. P. Vogels, Motor primitives in space and time via targeted gain modulation in cortical networks, Nat. Neurosci. 21, 1774 (2018).
  103. M. Mézard, The space of interactions in neural networks: Gardner’s computation with the cavity method, J. Phys. A 22, 2181 (1989).
  104. M. Mézard, Mean-field message-passing equations in the Hopfield model and its generalizations, Phys. Rev. E 95, 022117 (2017).
  105. F. Mastrogiuseppe, From dynamics to computations in recurrent neural networks, Ph.D. thesis, Université Paris sciences et lettres, 2017.

Outline

Information

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation