- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Neuronal correlations shape the scaling behavior of memory capacity and nonlinear computational capability of reservoir recurrent neural networks

Shotaro Takasu* and Toshio Aoyagi

  • Graduate School of Informatics, Kyoto University, Yoshida-Honmachi, Sakyo-ku, Kyoto 606-8501, Japan

  • *Contact author: takasu.shotaro.63x@st.kyoto-u.ac.jp
PDF

Phys. Rev. Research 7, 043083 – Published 21 October, 2025

DOI: https://doi.org/10.1103/cwvm-s53p

Abstract

Reservoir computing is a powerful framework for real-time information processing, characterized by its high computational ability and quick learning, with applications ranging from machine learning to biological systems. In this paper, we investigate how the computational ability of reservoir recurrent neural networks (RNNs) scales with an increasing number of readout neurons. First, we demonstrate that the memory capacity of a reservoir RNN scales sublinearly with the number of readout neurons. To elucidate this observation, we develop a theoretical framework for analytically deriving memory capacity that incorporates the effect of neuronal correlations, which have been ignored in prior theoretical work for analytical simplicity. Our theory successfully relates the sublinear scaling of memory capacity to the strength of neuronal correlations. Furthermore, we show this principle holds across diverse types of RNNs, even those beyond the direct applicability of our theory. Next, we numerically investigate the scaling behavior of nonlinear computational ability, which, alongside memory capacity, is crucial for overall computational performance. Our numerical simulations reveal that as memory capacity growth becomes sublinear, increasing the number of readout neurons successively enables nonlinear processing at progressively higher polynomial orders. Our theoretical framework suggests that neuronal correlations govern not only memory capacity but also the sequential growth of nonlinear computational capabilities. Our findings establish a foundation for designing scalable and cost-effective reservoir computing, providing insights into the interplay among neuronal correlations, linear memory, and nonlinear processing.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (68)

  1. I. D. Mienye and T. G. Swart, A comprehensive review of deep learning: Architectures, recent advances, and applications, Information 15, 755 (2024).
  2. S. S. Sengar, A. B. Hasan, S. Kumar, and F. Carroll, Generative artificial intelligence: A systematic review and applications, Multimedia Tools Appl. 84, 23661 (2025).
  3. G. Cybenko, Approximation by superpositions of a sigmoidal function, Math. Control Signal Syst. 2, 303 (1989).
  4. M. Raghu, B. Poole, J. Kleinberg, S. Ganguli, and J. Sohl-Dickstein, On the expressive power of deep neural networks, in Proceedings of the 34th International Conference on Machine Learning, in Proceedings of Machine Learning Research, edited by D. Precup and Y. W. Teh (PMLR, Cambridge, MA, 2017), Vol. 70, pp. 2847–2854.
  5. N. C. Thompson, K. Greenewald, K. Lee, and G. F. Manso, The computational limits of deep learning, arXiv:2007.05558.
  6. A. Marchisio, M. A. Hanif, F. Khalid, G. Plastiras, C. Kyrkou, T. Theocharides, and M. Shafique, Deep learning for edge computing: Current trends, cross-layer optimizations, and open research challenges, in 2019 IEEE Computer Society Annual Symposium on VLSI (ISVLSI) (IEEE Computer Society, New York, 2019), pp. 553–559.
  7. H. Jaeger, The “echo state” approach to analysing and training recurrent neural networks, Technical Report, GMD Report 148, German National Research Center for Information Technology, 2001.
  8. W. Maass, T. Natschlager, and H. Markram, Real-time computing without stable states: A new framework for neural computation based on perturbations, Neural Comput. 14, 2531 (2002).
  9. M. Yan, C. Huang, P. Bienstman, P. Tino, W. Lin, and J. Sun, Emerging opportunities and challenges for the future of reservoir computing, Nat. Commun. 15, 2056 (2024).
  10. K. Nakajima and I. Fischer, Reservoir Computing: Theory, Physical Implementations, and Applications (Springer, Berlin, 2021).
  11. M. Cucchi, S. Abreu, G. Ciccone, D. Brunner, and H. Kleemann, Hands-on reservoir computing: A tutorial for practical implementation, Neuromorph. Comput. Eng. 2, 032002 (2022).
  12. H. Jaeger, Short term memory in echo state networks, Technical Report, GMD Report 152, German National Research Center for Information Technology, 2002.
  13. J. Dambre, D. Verstraeten, B. Schrauwen, and S. Massar, Information processing capacity of dynamical systems, Sci. Rep. 2, 514 (2012).
  14. J. Schuecker, S. Goedeke, and M. Helias, Optimal sequence memory in driven random networks, Phys. Rev. X 8, 041029 (2018).
  15. T. Haruna and K. Nakajima, Optimal short-term memory before the edge of chaos in driven random recurrent networks, Phys. Rev. E 100, 062312 (2019).
  16. M. Lukoševičius, H. Jaeger, and B. Schrauwen, Reservoir computing trends, Künstliche Intell. 26, 365 (2012).
  17. D. Sussilo and L. F. Abbott, Generating coherent patterns of activity from chaotic neural networks, Neuron 63, 544 (2009).
  18. H. Sompolinsky, A. Crisanti, and H. J. Sommers, Chaos in random neural networks, Phys. Rev. Lett. 61, 259 (1988).
  19. L. Molgedey, J. Schuchhardt, and H. G. Schuster, Suppressing chaos in neural network by noise, Phys. Rev. Lett. 69, 3717 (1992).
  20. R. Engelken, A. Ingrosso, R. Khajeh, S. Goedeke, and L. F. Abbott, Input correlations impede suppression of chaos and learning in balanced firing-rate networks, PLoS Comput. Biol. 18, e1010590 (2022).
  21. S. Takasu and T. Aoyagi, Suppression of chaos in a partially driven recurrent neural network, Phys. Rev. Res. 6, 013172 (2024).
  22. M. Massar and M. Serge, Mean-field theory of echo state networks, Phys. Rev. E 87, 042809 (2013).
  23. K. Krishnamurthy, T. Can, and D. J. Schwab, Theory of gating in recurrent neural networks, Phys. Rev. X 12, 011011 (2022).
  24. T. Toyoizumi and L. F. Abbott, Beyond the edge of chaos: Amplification and temporal integration by recurrent networks in the chaotic regime, Phys. Rev. E 84, 051908 (2019).
  25. M. Helias and D. Dahmen, Statistical Field Theory for Neural Networks (Springer, Berlin, 2020).
  26. D. G. Clark, L. F. Abbott, and A. Litwin-Kumar, Dimension of activity in random neural networks, Phys. Rev. Lett. 131, 118401 (2023).
  27. M. Inubushi and K. Yoshimura, Reservoir computing beyond memory-nonlinearity trade-off, Sci. Rep. 7, 10199 (2017).
  28. G. Buzsáki and K. Mizuseki, The log-dynamic brain: How skewed distributions affect network operations, Nat. Rev. Neurosci. 15, 264 (2014).
  29. J.-n. Teramae, Y. Tsubo, and T. Fukai, Optimal spike-based communication in excitable networks with strong-sparse and weak-dense links, Sci. Rep. 2, 485 (2012).
  30. Ł. Kuśmierz, S. Ogawa, and T. Toyoizumi, Edge of chaos and avalanches in neural networks with heavy-tailed synaptic weight distribution, Phys. Rev. Lett. 125, 028101 (2020).
  31. A. Wardak and P. Gong, Extended Anderson criticality in heavy-tailed neural networks, Phys. Rev. Lett. 129, 048103 (2022).
  32. Y. Xie, S. Mihalas, and Łukasz Kuśmierz, Slow transition to low-dimensional chaos in heavy-tailed recurrent neural networks, arXiv:2505.09816.
  33. Y. Shao, D. Dahmen, S. Recanatesi, E. Shea-Brown, and S. Ostojic, Impact of local connectivity patterns on excitatory-inhibitory network dynamics, PRX Life 3, 023008 (2025).
  34. D. Martí, N. Brunel, and S. Ostojic, Correlations between synapses in pairs of neurons slow down dynamics in randomly connected neural networks, Phys. Rev. E 97, 062314 (2018).
  35. H. J. Sommers, A. Crisanti, H. Sompolinsky, and Y. Stein, Spectrum of large random asymmetric matrices, Phys. Rev. Lett. 60, 1895 (1988).
  36. B. Cramer, D. Stöckel, M. Kreft, M. Wibral, J. Schemmel, K. Meier, and V. Priesemann, Control of criticality and computation in spiking neuromorphic networks with plasticity, Nat. Commun. 11, 2853 (2020).
  37. A. Pikovsky and A. Politi, Lyapunov Exponents: A Tool to Explore Complex Dynamics (Cambridge University Press, Cambridge, UK, 2016).
  38. T. Hülser, F. Köster, K. Lüdge, and L. Jaurigue, Deriving task specific performance from the information processing capacity of a reservoir computer, Nanophotonics 12, 937 (2023).
  39. N. Bertschinger and T. Natschlager, Real-time computation at the edge of chaos in recurrent neural networks, Neural Comput. 16, 1413 (2004).
  40. R. Legenstein and W. Maass, Edge of chaos and prediction of computational performance for neural circuit models, Neural Networks 20, 323 (2007).
  41. T. L. Carroll, Do reservoir computers work best at the edge of chaos?, Chaos 30, 121109 (2020).
  42. J. J. Hopfield, Neural networks and physical systems with emergent collective computational abilities., Proc. Natl. Acad. Sci. USA 79, 2554 (1982).
  43. D. J. Amit, H. Gutfreund, and H. Sompolinsky, Storing infinite numbers of patterns in a spin-glass model of neural networks, Phys. Rev. Lett. 55, 1530 (1985).
  44. S. Amari, Characteristics of sparsely encoded associative memory, Neural Networks 2, 451 (1989).
  45. M. Okada, Notions of associative memory and sparse coding, Neural Networks 9, 1429 (1996).
  46. W. E. Vinje and J. L. Gallant, Sparse coding and decorrelation in primary visual cortex during natural vision, Science 287, 1273 (2000).
  47. C. van Vreeswijk and H. Sompolinsky, Chaos in neuronal networks with balanced excitatory and inhibitory activity, Science 274, 1724 (1996).
  48. A. Renart, J. de la Rocha, P. Bartho, L. Hollender, N. Parga, A. Reyes, and K. D. Harris, The asynchronous state in cortical circuits, Science 327, 587 (2010).
  49. D. Dahmen, S. Grün, M. Diesmann, and M. Helias, Second type of criticality in the brain uncovers rich multiple-neuron dynamics, Proc. Natl. Acad. Sci. USA 116, 13051 (2019).
  50. R. Rosenbaum, M. A. Smith, A. Kohn, J. E. Rubin, and B. Doiron, The spatial structure of correlated neuronal variability, Nat. Neurosci. 20, 107 (2017).
  51. S. Safavi, A. Dwarakanath, V. Kapoor, J. Werner, N. G. Hatsopoulos, N. K. Logothetis, and T. I. Panagiotaropoulos, Nonmonotonic spatial structure of interneuronal correlations in prefrontal microcircuits, Proc. Natl. Acad. Sci. USA 115, E3539 (2018).
  52. M. M. Churchland et al., Stimulus onset quenches neural variability: A widespread cortical phenomenon, Nat. Neurosci. 13, 369 (2010).
  53. A. Y. Y. Tan, Y. Chen, B. Scholl, E. Seidemann, and N. J. Priebe, Sensory stimulation shifts visual cortex from synchronous to asynchronous states, Nature (London) 509, 226 (2014).
  54. M. R. Cohen and J. H. R. Maunsell, Attention improves performance primarily by reducing interneuronal correlations, Nat. Neurosci. 12, 1594 (2009).
  55. J. F. Mitchell, K. A. Sundberg, and J. H. Reynolds, Spatial attention decorrelates intrinsic activity fluctuations in macaque area v4, Neuron 63, 879 (2009).
  56. S. Fusi, E. K. Miller, and M. Rigotti, Why neurons mix: High dimensionality for higher cognition, Curr. Opin. Neurobiol. 37, 66 (2016).
  57. C. Stringer, M. Pachitariu, N. Steinmetz, M. Carandini, and K. D. Harris, High-dimensional geometry of population responses in visual cortex, Nature (London) 571, 361 (2019).
  58. M. Farrell, S. Recanatesi, T. Moore, G. Lajoie, and E. Shea-Brown, Gradient-based learning drives robust representations in recurrent neural networks by balancing compression and expansion, Nat. Mach. Intell. 4, 564 (2022).
  59. K. Rajan, L. Abbott, and H. Sompolinsky, Inferring stimulus selectivity from the spatial structure of neural network dynamics, in Advances in Neural Information Processing Systems, edited by J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta (Curran Associates, Inc., Red Hook, NY, 2010), Vol. 23.
  60. A. Jacot, F. Gabriel, and C. Hongler, Neural tangent kernel: Convergence and generalization in neural networks, in Advances in Neural Information Processing Systems, edited by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Curran Associates, Inc., Red Hook, NY, 2018), Vol. 31.
  61. J. Lee, L. Xiao, S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Pennington, Wide neural networks of any depth evolve as linear models under gradient descent, in Advances in Neural Information Processing Systems, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett (Curran Associates, Inc., Red Hook, NY, 2019), Vol. 32.
  62. G. Yang and E. J. Hu, Tensor programs IV: Feature learning in infinite-width neural networks, in Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research, edited by M. Meila and T. Zhang (PMLR, Cambridge, MA, 2021), Vol. 139, pp. 11727–11737.
  63. B. Bordelon and C. Pehlevan, Self-consistent dynamical field theory of kernel evolution in wide neural networks, in Advances in Neural Information Processing Systems, edited by S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Curran Associates, Inc., Red Hook, NY, 2022), Vol. 35, pp. 32240–32256.
  64. K. Fischer, J. Lindner, D. Dahmen, Z. Ringel, M. Krämer, and M. Helias, Critical feature learning in deep neural networks, in Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, edited by R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (PMLR, Cambridge, MA, 2024), Vol. 235, pp. 13660–13690.
  65. S. Takasu, Source code for figures, GitHub, 2025, https://github.com/shtakasu/scalingofrc.git.
  66. W. Zou and H. Huang, Introduction to dynamical mean-field theory of randomly connected neural networks with bidirectionally correlated couplings, SciPost Phys. Lect. Notes 79 (2024).
  67. D. G. Clark, O. Marschall, A. van Meegen, and A. Litwin-Kumar, Connectivity structure and dynamics of nonlinear recurrent neural networks, arXiv:2409.01969.
  68. M. Dick, A. van Meegen, and M. Helias, Linking network- and neuron-level correlations by renormalized field theory, Phys. Rev. Res. 6, 033264 (2024).

Outline

Information

Phys. Rev. Research 7, 043083– Published 21 October, 2025
  • Received 29 April 2025
  • Accepted 22 September 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation