- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Entropy regularized reinforcement learning using large deviation theory

Argenis Arriojas1,*, Jacob Adamczyk1, Stas Tiomkin2, and Rahul V. Kulkarni1,†

  • 1Department of Physics, University of Massachusetts Boston, Boston, Massachusetts 02125, USA
  • 2Department of Computer Engineering, San Jose State University, San Jose, California 95192, USA

  • *arriojasmaldonado001@umb.edu
  • †rahul.kulkarni@umb.edu
PDF

Phys. Rev. Research 5, 023085 – Published 10 May, 2023

DOI: https://doi.org/10.1103/PhysRevResearch.5.023085

Abstract

Reinforcement learning (RL) is an important field of research in machine learning that is increasingly being applied to complex optimization problems in physics. In parallel, concepts from physics have contributed to important advances in RL with developments such as entropy-regularized RL. While these developments have led to advances in both fields, obtaining analytical solutions for optimization in entropy-regularized RL is currently an open problem. In this paper, we establish a mapping between entropy-regularized RL and research in nonequilibrium statistical mechanics focusing on Markovian processes conditioned on rare events. In the long-time limit, we apply approaches from large deviation theory to derive exact analytical results for the optimal policy and optimal dynamics in Markov decision process (MDP) models of reinforcement learning. The results obtained lead to an analytical and computational framework for entropy-regularized RL which is validated by simulations. The mapping established in this work connects current research in reinforcement learning and nonequilibrium statistical mechanics, thereby opening avenues for the application of analytical and computational approaches from one field to cutting-edge problems in the other.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (49)

  1. S. D. Sarma, D.-L. Deng, and L.-M. Duan, Machine learning meets quantum physics, Phys. Today 72(3), 48 (2019).
  2. P. Mehta, M. Bukov, C.-H. Wang, A. G. R. Day, C. Richardson, C. K. Fisher, and D. J. Schwab, A high-bias, low-variance introduction to machine learning for physicists, Phys. Rep. 810, 1 (2019).
  3. P. Mehta and D. J. Schwab, An exact mapping between the variational renormalization group and deep learning, arXiv:1410.3831.
  4. H. W. Lin, M. Tegmark, and D. Rolnick, Why does deep and cheap learning work so well?, J. Stat. Phys. 168, 1223 (2017).
  5. Y. Bahri, J. Kadmon, J. Pennington, S. S. Schoenholz, J. Sohl-Dickstein, and S. Ganguli, Statistical mechanics of deep learning, Annu. Rev. Condens. Matter Phys. 11, 501 (2020).
  6. M. Ostaszewski, L. M. Trenkwalder, W. Masarczyk, E. Scerri, and V. Dunjko, Reinforcement learning for optimization of variational quantum circuit architectures, Adv. Neural Inf. Process. Syst. 34, 18182 (2021).
  7. A. Barr, W. Gispen, and A. Lamacraft, Quantum ground states from reinforcement learning, in Proceedings of The First Mathematical and Scientific Machine Learning Conference Vol. 107 (PMLR, 2020), pp. 635–653, https://proceedings.mlr.press/v107/barr20a.html.
  8. A. Bolens and M. Heyl, Reinforcement Learning for Digital Quantum Simulation, Phys. Rev. Lett. 127, 110502 (2021).
  9. Y.-H. Zhang, P.-L. Zheng, Y. Zhang, and D.-L. Deng, Topological Quantum Compiling with Reinforcement Learning, Phys. Rev. Lett. 125, 170501 (2020).
  10. S. Borah, B. Sarma, M. Kewming, G. J. Milburn, and J. Twamley, Measurement-Based Feedback Quantum Control with Deep Reinforcement Learning for a Double-Well Nonlinear Potential, Phys. Rev. Lett. 127, 190403 (2021).
  11. J. Nousiainen, C. Rajani, M. Kasper, and T. Helin, Adaptive optics control using model-based reinforcement learning, Opt. Express 29, 15327 (2021).
  12. D. C. Rose, J. F. Mair, and J. P. Garrahan, A reinforcement learning approach to rare trajectory sampling, New J. Phys. 23, 013013 (2021).
  13. J. Yan and G. M. Rotskoff, Physics-informed graph neural networks enhance scalability of variational nonequilibrium optimal control, J. Chem. Phys. 157, 074101 (2022).
  14. S. Whitelam, D. Jacobson, and I. Tamblyn, Evolutionary reinforcement learning of dynamical large deviations, J. Chem. Phys. 153, 044113 (2020).
  15. S. Levine, Reinforcement learning and control as probabilistic inference: Tutorial and review, arXiv:1805.00909.
  16. H. J. Kappen, V. Gómez, and M. Opper, Optimal control as a graphical model inference problem, Mach. Learn. 87, 159 (2012).
  17. E. Todorov, General duality between optimal control and estimation, in 2008 47th IEEE Conference on Decision and Control (IEEE, Piscataway, NJ, 2008), pp. 4286–4292.
  18. K. Rawlik, M. Toussaint, and S. Vijayakumar, On stochastic optimal control and reinforcement learning by approximate inference, in Proceedings of Robotics: Science and Systems VIII (MIT Press, Cambridge, 2012).
  19. J. P. Garrahan, R. L. Jack, V. Lecomte, E. Pitard, K. van Duijvendijk, and F. van Wijland, First-order dynamical phase transition in models of glasses: an approach based on ensembles of histories, J. Phys. A: Math. Theor. 42, 075007 (2009).
  20. R. L. Jack and P. Sollich, Large deviations and ensembles of trajectories in stochastic models, Prog. Theor. Phys. Suppl. 184, 304 (2010).
  21. R. Chetrite and H. Touchette, Nonequilibrium Microcanonical and Canonical Ensembles and Their Equivalence, Phys. Rev. Lett. 111, 120601 (2013).
  22. R. Chetrite and H. Touchette, Nonequilibrium markov processes conditioned on large deviations, Ann. Henri Poincaré 16, 2005 (2015).
  23. R. Chetrite and H. Touchette, Variational and optimal control representations of conditioned and driven processes, J. Stat. Mech.: Theory Exp. (2015) P12001.
  24. H. D. Miller, A convexity property in the theory of random variables defined on a finite Markov chain, Ann. Math. Stat. 32, 1260 (1961).
  25. D. Simon, Construction of a coordinate Bethe ansatz for the asymmetric simple exclusion process with open boundaries, J. Stat. Mech.: Theory Exp. (2009) P07017.
  26. R. M. L. Evans, Rules for Transition Rates in Nonequilibrium Steady States, Phys. Rev. Lett. 92, 150601 (2004).
  27. R. M. L. Evans, Detailed balance has a counterpart in non-equilibrium steady states, J. Phys. A: Math. Gen. 38, 293 (2005).
  28. A. Simha, R. M. L. Evans, and A. Baule, Properties of a nonequilibrium heat bath, Phys. Rev. E 77, 031117 (2008).
  29. R. L. Jack and R. M. L. Evans, Absence of dissipation in trajectory ensembles biased by currents, J. Stat. Mech.: Theory Exp. (2016) 093305.
  30. A. Das, D. C. Rose, J. P. Garrahan, and D. T. Limmer, Reinforcement learning of rare diffusive dynamics, J. Chem. Phys. 155, 134105 (2021).
  31. L. C. G. Rogers and D. Williams, Diffusions, Markov Processes, and Martingales: Volume 1, Foundations, Cambridge Mathematical Library (Cambridge University Press, Cambridge, UK, 2000).
  32. D. A. Levin and Y. Peres, Markov Chains and Mixing Times, (American Mathematical Society, Providence, 2017).
  33. S. Méléard and D. Villemonais, Quasi-stationary distributions and population processes, Probab. Surv. 9, 340 (2012).
  34. E. A. van Doorn and P. K. Pollett, Quasi-stationary distributions for discrete-state models, Eur. J. Oper. Res. 230, 1 (2013).
  35. H. Touchette, A basic introduction to large deviations: Theory, applications, simulations, arXiv:1106.4146.
  36. E. Todorov, Linearly-solvable Markov decision problems, in Advances in Neural Information Processing Systems, Vol. 19, edited by B. Schölkopf, J. Platt, and T. Hoffman (MIT Press, Cambridge, 2007).
  37. E. Todorov, Efficient computation of optimal actions, Proc. Natl. Acad. Sci. USA 106, 11478 (2009).
  38. E. A. Theodorou, Nonlinear stochastic control and information theoretic dualities: Connections, interdependencies and thermodynamic interpretations, Entropy 17, 3352 (2015).
  39. V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, Playing atari with deep reinforcement learning, arXiv:1312.5602.
  40. C. Giardinà, J. Kurchan, and L. Peliti, Direct Evaluation of Large-Deviation Functions, Phys. Rev. Lett. 96, 120603 (2006).
  41. T. Nemoto, F. Bouchet, R. L. Jack, and V. Lecomte, Population-dynamics method with a multicanonical feedback control, Phys. Rev. E 93, 062123 (2016).
  42. J. Fu, K. Luo, and S. Levine, Learning robust rewards with adverserial inverse reinforcement learning, in International Conference on Learning Representations (2018), https://openreview.net/forum?id=rkHywl-A-.
  43. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction (MIT Press, Cambridge, 2018).
  44. V. S. Borkar, Learning algorithms for risk-sensitive control, in Proceedings of the 19th International Symposium on Mathematical Theory of Networks and Systems–MTNS, 2010 (unpublished).
  45. S. Levine, A. Kumar, G. Tucker, and J. Fu, Offline reinforcement learning: Tutorial, review, and perspectives on open problems, arXiv:2005.01643.
  46. R. Bellman, The theory of dynamic programming, Bull. Am. Math. Soc. 60, 503 (1954).
  47. G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, OpenAI Gym, arXiv:1606.01540.
  48. See Supplemental Material at http://link.aps.org/supplemental/10.1103/PhysRevResearch.5.023085 for example Python code to replicate results.
  49. https://github.com/argearriojas/2023-EntRegRL.

Outline

Information

Phys. Rev. Research 5, 023085– Published 10 May, 2023
  • Received 4 May 2021
  • Revised 8 June 2022
  • Accepted 1 April 2023
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation