- Physical Review Journals
All JournalsPhysics Magazine

Physical Review A

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Quantum framework for reinforcement learning: Integrating the Markov decision process, quantum arithmetic, and trajectory search

Thet Htar Su1,*, Shaswot Shresthamali2, and Masaaki Kondo1,3

  • *Contact author: thethtar@keio.jp; thethtarsu@acsl.ics.keio.ac.jp
PDF

Phys. Rev. A 111, 062421 – Published 17 June, 2025

DOI: https://doi.org/10.1103/5lfr-xb8m

Abstract

This paper introduces a quantum framework for addressing reinforcement learning (RL) tasks, grounded in the quantum principles and leveraging a fully quantum model of the classical Markov decision process. By employing quantum concepts and a quantum search algorithm, this work presents the implementation and optimization of the agent-environment interactions entirely within the quantum domain, eliminating reliance on classical computations. Key contributions include the quantum-based state transitions, return calculation, and trajectory search mechanism that utilize quantum principles to demonstrate the realization of RL processes through quantum phenomena. The implementation emphasizes the fundamental role of quantum superposition in enhancing computational efficiency for RL tasks. Results demonstrate the capacity of a quantum model to achieve quantum enhancement in RL, highlighting the potential of fully quantum implementations in decision-making tasks. This work not only underscores the applicability of quantum computing in machine learning but also contributes to the field of quantum reinforcement learning by offering a robust framework for understanding and exploiting quantum computing in RL systems.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (46)

  1. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction (The MIT Press, Cambridge, 2018).
  2. L. Graesser and W. Keng, Foundations of Deep Reinforcement Learning: Theory and Practice in Python (Addison-Wesley, Boston, 2020).
  3. I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning (The MIT Press, Cambridge, 2016).
  4. S. Shalev-Shwartz, S. Shammah, and A. Shashua, Safe, multi-agent, reinforcement learning for autonomous driving, arXiv:1610.03295.
  5. J. Kober, J. A. Bagnell, and J. Peters, Reinforcement learning in robotics: A survey, Int. J. Rob. Res. 32, 1238 (2013).
  6. D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. P. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis, Mastering the game of go without human knowledge, Nature (London) 550, 354 (2017).
  7. N. Brown and T. Sandholm, Superhuman AI for multiplayer poker, Science 365, 885 (2019).
  8. G. Dulac-Arnold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester, Challenges of real-world reinforcement learning: Definitions, benchmarks and analysis, Mach. Learn. 110, 2419 (2021).
  9. D. Silver, A. Huang, C. Maddison, A. Guez, L. Sifre, G. Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, Mastering the game of go with deep neural networks and tree search, Nature (London) 529, 484 (2016).
  10. T. L. Scholten, C. J. Williams, D. Moody, M. Mosca, W. Hurley, W. J. Zeng, M. Troyer, and J. M. Gambetta, Assessing the benefits and risks of quantum computers, arXiv:2401.16317.
  11. N. Meyer, C. Ufrecht, M. Periyasamy, D. D. Scherer, A. Plinge, and C. Mutschler, A survey on quantum reinforcement learning, arXiv:2211.03464.
  12. L. K. Grover, A fast quantum mechanical algorithm for database search, in Proceedings of the Twenty-Eighth Annual ACM Symposium on Theory of Computing (ACM Press, New York, 1996), pp. 212–219.
  13. D. Dong, C. Chen, H. Li, and T.-J. Tarn, Quantum reinforcement learning, IEEE Trans. Syst. Man Cybern. Part B Cybern. 38, 1207 (2008).
  14. D. Dao-Yi, C. Chun-Lin, C. Zong-Hai, and Z. Chen-Bin, Quantum mechanics helps in learning for more intelligent robots, Chin. Phys. Lett. 23, 1691 (2006).
  15. C.-L. Chen and D.-Y. Dong, Superposition-inspired reinforcement learning and quantum reinforcement learning, in Reinforcement Learning, edited by C. Weber, M. Elshaw, and N. M. Mayer (IntechOpen, Rijeka, 2008), Chap. 4.
  16. C. L. CHEN, D. Y. DONG, and Z. H. CHEN, Quantum computation for action selection using reinforcement learning, Int. J. Quantum Inf. 04, 1071 (2006).
  17. D. Dong, C. Chen, J. Chu, and T.-J. Tarn, Robust quantum-inspired reinforcement learning for robot navigation, IEEE/ASME Trans. Mechatron. 17, 86 (2012).
  18. M. Ganger and W. Hu, Quantum multiple Q-learning, Int. J. Intell. Sci. 09, 1 (2019).
  19. B. Cho, Y. Xiao, P. Hui, and D. Dong, Quantum bandit with amplitude amplification exploration in an adversarial environment, IEEE Trans. Knowl. Data Eng. 36, 311 (2024).
  20. Q. Wei, H. Ma, C. Chen, and D. Dong, Deep reinforcement learning with quantum-inspired experience replay, IEEE Trans. Cybern. 52, 9326 (2022).
  21. Y. Li, A. H. Aghvami, and D. Dong, Intelligent trajectory planning in UAV-mounted wireless networks: A quantum-inspired reinforcement learning perspective, IEEE Wireless Commun. Lett. 10, 1994 (2021).
  22. J.-A. Li, D. Dong, Z. Wei, Y. Liu, Y. Pan, F. Nori, and X. Zhang, Quantum reinforcement learning during human decision-making, Nat. Hum. Behav. 4, 294 (2020).
  23. D. Niraula, J. Jamaluddin, M. M. Matuszak, R. K. T. Haken, and I. E. Naqa, Quantum deep reinforcement learning for clinical decision support in oncology: Application to adaptive radiotherapy, Sci. Rep. 11, 23545 (2021).
  24. A. Sequeira, L. P. Santos, and L. S. Barbosa, Policy gradients using variational quantum circuits, Quantum Mach. Intell. 5, 18 (2023).
  25. S. Y.-C. Chen, C.-H. H. Yang, J. Qi, P.-Y. Chen, X. Ma, and H.-S. Goan, Variational quantum circuits for deep reinforcement learning, IEEE Access 8, 141007 (2020).
  26. O. Lockwood and M. Si, Reinforcement learning with quantum variational circuits, in Proceedings of the Sixteenth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, AIIDE'20 (AAAI Press, Washington, DC, 2020), Vol. 16, pp. 245-251.
  27. O. Lockwood and M. Si, Playing Atari with hybrid quantum-classical reinforcement learning, in NeurIPS 2020 Workshop on Pre-registration in Machine Learning (PMLR, USA, 2021), Vol. 148, pp. 285–301.
  28. S. Wu, S. Jin, D. Wen, D. Han, and X. Wang, Quantum reinforcement learning in continuous action space, Quantum 9, 1660 (2025).
  29. A. Skolik, S. Jerbi, and V. Dunjko, Quantum agents in the Gym: A variational quantum algorithm for deep Q-learning, Quantum 6, 720 (2022).
  30. S. Jerbi, C. Gyurik, S. C. Marshall, H. J. Briegel, and V. Dunjko, Parametrized quantum policies for reinforcement learning, in Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS'21 (Curran Associates, Red Hook, NY, 2021), pp. 28362–28375.
  31. Y. Kwak, W. J. Yun, S. Jung, J.-K. Kim, and J. Kim, Introduction to quantum reinforcement learning: Theory and pennylane-based implementation, in Proceedings of the 2021 International Conference on Information and Communication Technology Convergence (ICTC) (IEEE, Korea, 2021), pp. 416–420.
  32. Q. Lan, Variational quantum soft actor-critic, arXiv:2112.11921.
  33. D. Wang, A. Sundaram, R. Kothari, A. Kapoor, and M. Roetteler, Quantum algorithms for reinforcement learning with a generative model, in Proceedings of the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research Vol. 139, edited by M. Meila and T. Zhang (PMLR, 2021), pp. 10916–10926.
  34. E. A. Cherrat, I. Kerenidis, and A. Prakash, Quantum reinforcement learning via policy iteration, Quantum Mach. Intell. 5, 30 (2023).
  35. S. Wiedemann, D. Hein, S. Udluft, and C. Mendl, Quantum policy iteration via amplitude estimation and grover search–towards quantum advantage for reinforcement learning, in Transactions on Machine Learning Research (2023).
  36. V. Dunjko, J. M. Taylor, and H. J. Briegel, Quantum-enhanced machine learning, Phys. Rev. Lett. 117, 130501 (2016).
  37. A. Plaat, Deep Reinforcement Learning (Springer Nature, Singapore, 2022).
  38. M. Morales, Grokking Deep Reinforcement Learning (Manning, New York, 2020).
  39. E. Rieffel and W. Polak, Quantum Computing: A Gentle Introduction (The MIT Press, Cambridge, 2011).
  40. M. A. Nielsen and I. L. Chuang, Quantum Computation and Quantum Information (Cambridge University Press, Cambridge, UK, 2011).
  41. P. W. Shor, Algorithms for quantum computation: Discrete logarithms and factoring, in Proceedings of the 35th Annual Symposium on Foundations of Computer Science (IEEE, Piscataway, NJ, 1994), pp. 124–134.
  42. P. W. Shor, Polynomial-time algorithms for prime factorization and discrete logarithms on a quantum computer, SIAM J. Comput. 26, 1484 (1997).
  43. A. Ekert and R. Jozsa, Quantum computation and Shor's factoring algorithm, Rev. Mod. Phys. 68, 733 (1996).
  44. A. Javadi-Abhari, M. Treinish, K. Krsulich, C. J. Wood, J. Lishman, J. Gacon, S. Martiel, P. D. Nation, L. S. Bishop, A. W. Cross, B. R. Johnson, and J. M. Gambetta, Quantum computing with Qiskit, arXiv:2405.08810.
  45. P. Kaye, R. Laflamme, and M. Mosca, An Introduction to Quantum Computing (Oxford University Press, New York, 2007).
  46. C. Guo, Grover's algorithm–Implementations and implications, Highlights Sci., Eng. Technol. 38, 1071 (2023).

Outline

Information

Phys. Rev. A 111, 062421– Published 17 June, 2025
  • Received 23 December 2024
  • Accepted 22 May 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation