- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Letter
  • Open Access

Reservoir computing with diverse timescales for prediction of multiscale dynamics

Gouhei Tanaka1,2,3, Tadayoshi Matsumori4, Hiroaki Yoshida4, and Kazuyuki Aihara1

  • 1International Research Center for Neurointelligence, The University of Tokyo, Tokyo 113-0033, Japan
  • 2Department of Electrical Engineering and Information Systems, Graduate School of Engineering, The University of Tokyo, Tokyo 113-8656, Japan
  • 3Department of Mathematical Informatics, Graduate School of Information Science and Technology, The University of Tokyo, Tokyo 113-8656, Japan
  • 4Toyota Central R&D Labs., Inc., Tokyo 112-0004, Japan

PDF

Phys. Rev. Research 4, L032014 – Published 28 July, 2022

DOI: https://doi.org/10.1103/PhysRevResearch.4.L032014

Abstract

Machine learning approaches have recently been leveraged as a substitute or an aid for physical/mathematical modeling approaches to dynamical systems. To develop an efficient machine learning method dedicated to modeling and prediction of multiscale dynamics, we propose a reservoir computing (RC) model with diverse timescales by using a recurrent network of heterogeneous leaky integrator (LI) neurons. We evaluate computational performance of the proposed model in two time series prediction tasks related to four chaotic fast-slow dynamical systems. In a one-step-ahead prediction task where input data are provided only from the fast subsystem, we show that the proposed model yields better performance than the standard RC model with identical LI neurons. Our analysis reveals that the timescale required for producing each component of target multiscale dynamics is appropriately and flexibly selected from the reservoir dynamics by model training. In a long-term prediction task, we demonstrate that a closed-loop version of the proposed model can achieve longer-term predictions compared to the counterpart with identical LI neurons depending on the hyperparameter setting.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (42)

  1. D. G. Vlachos, A review of multiscale analysis: examples from systems biology, materials engineering, and other fluid–surface interacting systems, Adv. Chem. Eng. 30, 1 (2005).
  2. K. Matouš, M. G. Geers, V. G. Kouznetsova, and A. Gillman, A review of predictive nonlinear theories for multiscale modeling of heterogeneous materials, J. Comput. Phys. 330, 192 (2017).
  3. T. S. Deisboeck, Z. Wang, P. Macklin, and V. Cristini, Multiscale cancer modeling, Annu. Rev. Biomed. Eng. 13, 127 (2011).
  4. J. Walpole, J. A. Papin, and S. M. Peirce, Multiscale computational models of complex biological systems, Annu. Rev. Biomed. Eng. 15, 137 (2013).
  5. E. Weinan, Principles of Multiscale Modeling (Cambridge University Press, 2011).
  6. J. Pathak, B. Hunt, M. Girvan, Z. Lu, and E. Ott, Model-Free Prediction of Large Spatiotemporally Chaotic Systems from Data: A Reservoir Computing Approach, Phys. Rev. Lett. 120, 024102 (2018).
  7. J. Pathak, A. Wikner, R. Fussell, S. Chandra, B. R. Hunt, M. Girvan, and E. Ott, Hybrid forecasting of chaotic processes: Using machine learning in conjunction with a knowledge-based model, Chaos 28, 041101 (2018).
  8. M. Alber, A. B. Tepole, W. R. Cannon, S. De, S. Dura-Bernal, K. Garikipati, G. Karniadakis, W. W. Lytton, P. Perdikaris, L. Petzold et al., Integrating machine learning and multiscale modeling perspectives, challenges, and opportunities in the biological, biomedical, and behavioral sciences, npj Digital Medicine 2, 115 (2019).
  9. H. Jaeger, The “echo state” approach to analysing and training recurrent neural networks, Bonn, Germany: German National Research Center for Information Technology GMD Technical Report 148 (2001).
  10. H. Jaeger and H. Haas, Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication, Science 304, 78 (2004).
  11. W. Maass, T. Natschläger, and H. Markram, Real-time computing without stable states: A new framework for neural computation based on perturbations, Neural Comput. 14, 2531 (2002).
  12. D. Verstraeten, B. Schrauwen, M. D'Haene, and D. Stroobandt, An experimental unification of reservoir computing methods, Neural Networks 20, 391 (2007).
  13. M. Lukoševičius and H. Jaeger, Reservoir computing approaches to recurrent neural network training, Comput. Sci. Rev. 3, 127 (2009).
  14. P. J. Werbos, Backpropagation through time: what it does and how to do it, Proc. IEEE 78, 1550 (1990).
  15. M. Lukoševičius, A practical guide to applying echo state networks, in Neural Networks: Tricks of the Trade (Springer, 2012), pp. 659–686.
  16. H. Jaeger, M. Lukoševičius, D. Popovici, and U. Siewert, Optimization and applications of echo state networks with leaky-integrator neurons, Neural Networks 20, 335 (2007).
  17. H. Jaeger, Discovering multiscale dynamical features with hierarchical echo state networks, Tech. Rep. (Jacobs University Bremen, 2007).
  18. L. Manneschi, M. O. Ellis, G. Gigante, A. C. Lin, P. Del Giudice, and E. Vasilaki, Exploiting multiple timescales in hierarchical echo state networks, Front. Appl. Math. Stat. 6, 616658 (2021).
  19. G. Tanaka, T. Yamane, J. B. Héroux, R. Nakane, N. Kanazawa, S. Takeda, H. Numata, D. Nakano, and A. Hirose, Recent advances in physical reservoir computing: A review, Neural Networks 115, 100 (2019).
  20. C. Gallicchio, A. Micheli, and L. Pedrelli, Deep reservoir computing: A critical experimental analysis, Neurocomputing 268, 87 (2017).
  21. H. Tamura and G. Tanaka, Transfer-RLS method and transfer-FORCE learning for simple and fast training of reservoir computing models, Neural Networks 143, 550 (2021).
  22. Z. Li and G. Tanaka, Multi-reservoir echo state networks with sequence resampling for nonlinear time-series prediction, Neurocomputing 467, 115 (2022).
  23. T. Akiyama and G. Tanaka, Computational efficiency of multi-step learning echo state networks for nonlinear time series prediction, IEEE Access 10, 28535 (2022).
  24. M. Di Volo and A. Destexhe, Optimal responsiveness and information flow in networks of heterogeneous neurons, Sci. Rep. 11, 17611 (2021).
  25. G. Tanaka, R. Nakane, T. Yamane, D. Nakano, S. Takeda, S. Nakagawa, and A. Hirose, Exploiting heterogeneous units for reservoir computing with simple architecture, in International Conference on Neural Information Processing (Springer, 2016), pp. 187–194.
  26. L. Appeltant, M. C. Soriano, G. Van der Sande, J. Danckaert, S. Massar, J. Dambre, B. Schrauwen, C. R. Mirasso, and I. Fischer, Information processing using a single dynamical node as complex system, Nat. Commun. 2, 468 (2011).
  27. Y. Paquot, F. Duport, A. Smerieri, J. Dambre, B. Schrauwen, M. Haelterman, and S. Massar, Optoelectronic reservoir computing, Sci. Rep. 2, 287 (2012).
  28. F. Stelzer, A. Röhm, K. Lüdge, and S. Yanchuk, Performance boost of time-delay reservoir computing by non-resonant clock cycle, Neural Networks 124, 158 (2020).
  29. J. Pauwels, G. Verschaffelt, S. Massar, and G. Van der Sande, Distributed Kerr non-linearity in a coherent all-optical fiber-ring reservoir computer, Front. Phys. 7, 138 (2019).
  30. A. K. Seshadri, Fast–slow climate dynamics and peak global warming, Climate Dynamics 48, 2235 (2017).
  31. C. S. Meinen, R. C. Perez, S. Dong, A. R. Piola, and E. Campos, Observed ocean bottom temperature variability at four sites in the northwestern Argentine Basin: evidence of decadal deep/abyssal warming amidst hourly to interannual variability during 2009–2019, Geophys. Res. Lett. 47, e2020GL089093 (2020).
  32. N. F. Rulkov, Regularization of Synchronized Chaotic Bursts, Phys. Rev. Lett. 86, 183 (2001).
  33. J. L. Hindmarsh and R. Rose, A model of neuronal bursting using three coupled first order differential equations, Proc. R. Soc. London B 221, 87 (1984).
  34. K. P. Champion, S. L. Brunton, and J. N. Kutz, Discovery of nonlinear multiscale systems: Sampling strategies and embeddings, SIAM J. Appl. Dynam. Syst. 18, 312 (2019).
  35. G. Boffetta, P. Giuliani, G. Paladin, and A. Vulpiani, An extension of the Lyapunov analysis for the predictability problem, J. Atmos. Sci. 55, 3409 (1998).
  36. See Supplemental Material at http://link.aps.org/supplemental/10.1103/PhysRevResearch.4.L032014 for further details.
  37. F. Borra, A. Vulpiani, and M. Cencini, Effective models and predictability of chaotic multiscale systems via machine learning, Phys. Rev. E 102, 052203 (2020).
  38. N. Kashtan and U. Alon, Spontaneous evolution of modularity and network motifs, Proc. Natl. Acad. Sci. 102, 13773 (2005).
  39. D. M. Lorenz, A. Jeng, and M. W. Deem, The emergence of modularity in biological systems, Phys. Life Rev. 8, 129 (2011).
  40. G. R. Yang, M. R. Joglekar, H. F. Song, W. T. Newsome, and X.-J. Wang, Task representations in neural networks trained to perform many cognitive tasks, Nat. Neurosci. 22, 297 (2019).
  41. P. R. Vlachas, J. Pathak, B. R. Hunt, T. P. Sapsis, M. Girvan, E. Ott, and P. Koumoutsakos, Backpropagation algorithms and reservoir computing in recurrent neural networks for the forecasting of complex spatiotemporal dynamics, Neural Networks 126, 191 (2020).
  42. T. Proix, V. K. Jirsa, F. Bartolomei, M. Guye, and W. Truccolo, Predicting the spatiotemporal diversity of seizure propagation and termination in human focal epilepsy, Nat. Commun. 9, 1088 (2018).

Outline

Information

Phys. Rev. Research 4, L032014– Published 28 July, 2022
  • Received 23 August 2021
  • Accepted 28 June 2022
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation