- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Letter
  • Open Access

Embedding theory of reservoir computing and reducing reservoir network using time delays

Xing-Yue Duan1, Xiong Ying2, Si-Yang Leng3,4,5, Jürgen Kurths3,6, Wei Lin2,3,5,7,*, and Huan-Fei Ma1,†

  • 1School of Mathematical Sciences, Soochow University, Suzhou 215006, China
  • 2School of Mathematical Sciences, SCMS, and SCAM, Fudan University, Shanghai 200433, China
  • 3Research Institute of Intelligent Complex Systems and Centre for Computational Systems Biology, Fudan University, Shanghai 200433, China
  • 4Academy for Engineering and Technology, Fudan University, Shanghai 200433, China
  • 5State Key Laboratory of Medical Neurobiology, LCNBI, and MOE Frontiers Center for Brain Science, Institutes of Brain Science, Fudan University, Shanghai 200032, China
  • 6Potsdam Institute for Climate Impact Research (PIK), Potsdam 14473, Germany
  • 7Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China

  • *wlin@fudan.edu.cn
  • †hfma@suda.edu.cn
PDF

Phys. Rev. Research 5, L022041 – Published 25 May, 2023

DOI: https://doi.org/10.1103/PhysRevResearch.5.L022041

Abstract

Reservoir computing (RC), a particular form of recurrent neural network, is under explosive development due to its exceptional efficacy and high performance in reconstruction and/or prediction of complex physical systems. However, the mechanism triggering such effective applications of RC is still unclear, awaiting deep and systematic exploration. Here, combining the delayed embedding theory with the generalized embedding theory, we rigorously prove that RC is essentially a high-dimensional embedding of the original input nonlinear dynamical system. Thus, using this embedding property, we unify into a universal framework the standard RC and the time-delayed RC where we introduce time delays only into the network's output layer, and we further find a trade-off relation between the time delays and the number of neurons in RC. Based on these findings, we significantly reduce the RC's network size and promote its memory capacity in completing systems reconstruction and prediction. More surprisingly, only using a single-neuron reservoir with time delays is sometimes sufficient for achieving reconstruction and prediction tasks, while the standard RC of any large size but without time delay cannot complete them yet.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (37)

  1. H. Jaeger and H. Haas, Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication, Science 304, 78 (2004).
  2. H. Jaeger, The “echo state” approach to analysing and training recurrent neural networks – with an erratum note, GMD Technical Report No. 148, German National Research Center for Information Technology, Bonn, 2001.
  3. W. Maass, T. Natschläger, and H. Markram, Real-time computing without stable states: A new framework for neural computation based on perturbations, Neural Comput. 14, 2531 (2002).
  4. J. Pathak, B. Hunt, M. Girvan, Z. Lu, and E. Ott, Model-Free Prediction of Large Spatiotemporally Chaotic Systems from Data: A Reservoir Computing Approach, Phys. Rev. Lett. 120, 024102 (2018).
  5. Q. Zhu, H. Ma, and W. Lin, Detecting unstable periodic orbits based only on time series: When adaptive delayed feedback control meets reservoir computing, Chaos 29, 093125 (2019).
  6. G. Tanaka, T. Yamane, J. B. Héroux, R. Nakane, N. Kanazawa, S. Takeda, H. Numata, D. Nakano, and A. Hirose, Recent advances in physical reservoir computing: A review, Neural Networks 115, 100 (2019).
  7. Y. Tang, J. Kurths, W. Lin, E. Ott, and L. Kocarev, Introduction to focus issue: When machine learning meets complex systems: Networks, chaos, and nonlinear dynamics, Chaos 30, 063151 (2020).
  8. H. Jaeger, Tutorial on Training Recurrent Neural Networks, Covering BPPT, RTRL, EKF and the ‘Echo State Network” Approach (GMD-Forschungszentrum Informationstechnik, Bonn, 2002), Vol. 5.
  9. A. Haluszczynski and C. Räth, Good and bad predictions: Assessing and improving the replication of chaotic attractors by means of reservoir computing, Chaos 29, 103143 (2019).
  10. A. Griffith, A. Pomerance, and D. J. Gauthier, Forecasting chaotic systems with very low connectivity reservoir computers, Chaos 29, 123108 (2019).
  11. M. Lukoševičius and H. Jaeger, Reservoir computing approaches to recurrent neural network training, Comput. Sci. Rev. 3, 127 (2009).
  12. J. Jiang and Y.-C. Lai, Model-free prediction of spatiotemporal dynamical systems with recurrent neural networks: Role of network spectral radius, Phys. Rev. Res. 1, 033056 (2019).
  13. J. Boedecker, O. Obst, J. T. Lizier, N. M. Mayer, and M. Asada, Information processing in echo state networks at the edge of chaos, Theory Biosci. 131, 205 (2012).
  14. J. Dambre, D. Verstraeten, B. Schrauwen, and S. Massar, Information processing capacity of dynamical systems, Sci. Rep. 2, 514 (2012).
  15. I. B. Yildiz, H. Jaeger, and S. J. Kiebel, Re-visiting the echo state property, Neural Networks 35, 1 (2012).
  16. G. Manjunath and H. Jaeger, Echo state property linked to an input: Exploring a fundamental characteristic of recurrent neural networks, Neural Comput. 25, 671 (2013).
  17. H. Chang, S. Nakaoka, and H. Ando, Effect of shapes of activation functions on predictability in the echo state network, arXiv:1905.09419.
  18. J. J. Steil, Online reservoir adaptation by intrinsic plasticity for backpropagation–decorrelation and echo state learning, Neural Networks 20, 353 (2007).
  19. E. Bollt, On explaining the surprising success of reservoir computing forecaster of chaos? The universal machine learning dynamical system with contrast to VAR and DMD, Chaos 31, 013108 (2021).
  20. D. J. Gauthier, E. Bollt, A. Griffith, and W. A. S. Barbosa, Next generation reservoir computing, Nat. Commun. 12, 5564 (2021).
  21. A. Hart, J. Hook, and J. Dawes, Embedding and approximation theorems for echo state networks, Neural Networks 128, 234 (2020).
  22. A. Hart, J. Hook, and J. Dawes, Echo state networks trained by Tikhonov least squares are l2 (μ) approximators of ergodic dynamical systems, Physica D 421, 132882 (2021).
  23. L. Appeltant, M. C. Soriano, G. Van der Sande, J. Danckaert, S. Massar, J. Dambre, B. Schrauwen, C. R. Mirasso, and I. Fischer, Information processing using a single dynamical node as complex system, Nat. Commun. 2, 468 (2011).
  24. D. Brunner, B. Penkovsky, B. A. Marquez, M. Jacquot, I. Fischer, and L. Larger, Tutorial: Photonic neural networks in delay systems, J. Appl. Phys. 124, 152004 (2018).
  25. L. Larger, A. Baylón-Fuentes, R. Martinenghi, V. S. Udaltsov, Y. K. Chembo, and M. Jacquot, High-Speed Photonic Reservoir Computing Using a Time-Delay-Based Architecture: Million Words per Second Classification, Phys. Rev. X 7, 011015 (2017).
  26. See Supplemental Material at http://link.aps.org/supplemental/10.1103/PhysRevResearch.5.L022041 for details of the proof of the embedding theory, the system models, the parameters settings, and the comparison study with the photonic reservoir.
  27. F. Takens, Detecting strange attractors in turbulence, Lect. Notes Math. 898, 366 (1981).
  28. T. Sauer, J. Yorke, and M. Casdagli, Embedology, J. Stat. Phys. 65, 579 (1991).
  29. H.-F. Ma, S. Leng, K. Aihara, W. Lin, and L. Chen, Randomly distributed embedding making short-term high-dimensional data predictable, Proc. Natl. Acad. Sci. 115, E9994 (2018).
  30. J.-W. Hou, H.-F. Ma, D. He, J. Sun, Q. Nie, and W. Lin, Harvesting random embedding for high-frequency change-point detection in temporal complex systems, Natl. Sci. Rev. 9, nwab228 (2022).
  31. Y. Suzuki, M. Lu, E. Ben-Jacob, and J. N. Onuchic, Periodic, quasi-periodic and chaotic dynamics in simple gene elements with time delays, Sci. Rep. 6, 21037 (2016).
  32. H. Wernecke, B. Sándor, and C. Gros, Chaos in time delay systems, an educational review, Phys. Rep. 824, 1 (2019).
  33. A. W. M. Dress, W. Hordijk, W. Lin, and P. Serocka, The ideal storage cellular automaton model, in Structure Discovery in Biology: Motifs, Networks & Phylogenies in Dagstuhl Seminar Proceedings (Schloss Dagstuhl-Leibniz-Zentrum für Informatik, Wadern, Germany, 2010).
  34. A. W. Dress and W. Lin, Dynamics of a discrete-time model of an “ideal-storage” system describing hetero-catalytic processes on metal surfaces, I. J. Bifurcation Chaos 21, 1331 (2011).
  35. M. J. McGuinness, The fractal dimension of the Lorenz attractor, Phys. Lett. A 99, 5 (1983).
  36. D. Viswanath, The fractal property of the Lorenz attractor, Physica D 190, 115 (2004).
  37. Y. Sakemi, K. Morino, T. Leleu, and K. Aihara, Model-size reduction for reservoir computing by concatenating internal states through time, Sci. Rep. 10, 21794 (2020).

Outline

Information

Phys. Rev. Research 5, L022041– Published 25 May, 2023
  • Received 5 November 2021
  • Revised 16 March 2023
  • Accepted 8 May 2023
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation