- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Letter
  • Open Access

Pareto-optimal cycles for power, efficiency and fluctuations of quantum heat engines using reinforcement learning

Paolo A. Erdman1, Alberto Rolandi2, Paolo Abiuso3, Martí Perarnau-Llobet2,*, and Frank Noé6,1,4,5,†

  • 1Department of Mathematics and Computer Science, Freie Universität Berlin, Arnimallee 6, 14195 Berlin, Germany
  • 2Department of Applied Physics, University of Geneva, 1211 Geneva, Switzerland
  • 3ICFO Institut de Ciències Fotòniques, The Barcelona Institute of Science and Technology, 08860 Castelldefels (Barcelona), Spain
  • 4Department of Physics, Freie Universität Berlin, Arnimallee 6, 14195 Berlin, Germany
  • 5Department of Chemistry, Rice University, Houston, Texas 77005, USA
  • 6Microsoft Research AI4Science, Karl-Liebknecht Str. 32, 10178 Berlin, Germany

  • *marti.perarnaullobet@unige.ch
  • †frank.noe@fu-berlin.de
PDF

Phys. Rev. Research 5, L022017 – Published 27 April, 2023

DOI: https://doi.org/10.1103/PhysRevResearch.5.L022017

Abstract

The full optimization of a quantum heat engine requires operating at high power, high efficiency, and high stability (i.e., low power fluctuations). However, these three objectives cannot be simultaneously optimized—as indicated by the so-called thermodynamic uncertainty relations—and a systematic approach to finding optimal balances between them including power fluctuations has, as yet, been elusive. Here we propose such a general framework to identify Pareto-optimal cycles for driven quantum heat engines that trade off power, efficiency, and fluctuations. We then employ reinforcement learning to identify the Pareto front of a quantum dot-based engine and find abrupt changes in the form of optimal cycles when switching between optimizing two and three objectives. We further derive analytical results in the fast- and slow-driving regimes that accurately describe different regions of the Pareto front.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (117)

  1. F. Giazotto, T. T. Heikkilä, A. Luukanen, A. M. Savin, and J. P. Pekola, Opportunities for mesoscopics in thermometry and refrigeration: Physics and applications, Rev. Mod. Phys. 78, 217 (2006).
  2. J. P. Pekola, Towards quantum thermodynamics in electronic circuits, Nat. Phys. 11, 118 (2015).
  3. S. Vinjanampathy and J. Anders, Quantum thermodynamics, Contemp. Phys. 57, 545 (2016).
  4. G. Benenti, G. Casati, K. Saito, and R. S.Whitney, Fundamental aspects of steady-state conversion of heat to work at the nanoscale, Phys. Rep. 694, 1 (2017).
  5. F. Binder, L. Correa, C. Gogolin, J. Anders, and G. Adesso, Thermodynamics in the Quantum Regime: Fundamental Aspects and New Directions (Springer International Publishing, Cham, Switzerland, 2019).
  6. E. Geva and R. Kosloff, A quantum-mechanical heat engine operating in finite time. a model consisting of spin-1/2 systems as the working fluid, J. Chem. Phys. 96, 3054 (1992).
  7. T. Feldmann and R. Kosloff, Performance of discrete heat engines and heat pumps in finite time, Phys. Rev. E 61, 4774 (2000).
  8. T. Schmiedl and U. Seifert, Efficiency at maximum power: An analytically solvable model for stochastic heat engines, Europhys. Lett. 81, 20003 (2008).
  9. M. Esposito, R. Kawai, K. Lindenberg, and C. V. den Broeck, Efficiency at Maximum Power of Low-Dissipation Carnot Engines, Phys. Rev. Lett. 105, 150603 (2010).
  10. M. Esposito, R. Kawai, K. Lindenberg, and C. Van den Broeck, Quantum-dot carnot engine at maximum power, Phys. Rev. E 81, 041106 (2010).
  11. A. E. Allahverdyan, K. V. Hovhannisyan, A. V. Melkikh, and S. G. Gevorkian, Carnot Cycle at Finite Power: Attainability of Maximal Efficiency, Phys. Rev. Lett. 111, 050601 (2013).
  12. S. Juergens, F. Haupt, M. Moskalets, and J. Splettstoesser, Thermoelectric performance of a driven double quantum dot, Phys. Rev. B 87, 245423 (2013).
  13. V. Holubec and A. Ryabov, Efficiency at and near maximum power of low-dissipation heat engines, Phys. Rev. E 92, 052125 (2015).
  14. V. Holubec and A. Ryabov, Maximum efficiency of low-dissipation heat engines at arbitrary power, J. Stat. Mech.: Theory Exp. (2016) 073204.
  15. B. Karimi and J. P. Pekola, Otto refrigerator based on a superconducting qubit: Classical and quantum performance, Phys. Rev. B 94, 184503 (2016).
  16. M. Campisi and R. Fazio, The power of a critical heat engine, Nat. Commun. 7, 11895 (2016).
  17. K. Yamamoto, O. Entin-Wohlman, A. Aharony, and N. Hatano, Efficiency bounds on thermoelectric transport in magnetic fields: The role of inelastic processes, Phys. Rev. B 94, 121402(R) (2016).
  18. R. Kosloff and Y. Rezek, The quantum harmonic otto cycle, Entropy 19, 136 (2017).
  19. V. Patel, V. Savsani, and A. Mudgal, Many-objective thermodynamic optimization of Stirling heat engine, Energy 125, 629 (2017).
  20. V. Cavina, A. Mari, A. Carlini, and V. Giovannetti, Optimal thermodynamic control in open quantum systems, Phys. Rev. A 98, 012139 (2018).
  21. P. Abiuso and V. Giovannetti, Non-markov enhancement of maximum power for quantum thermal machines, Phys. Rev. A 99, 052106 (2019).
  22. Y.-H. Ma, D. Xu, H. Dong, and C.-P. Sun, Universal constraint for efficiency and power of a low-dissipation heat engine, Phys. Rev. E 98, 042112 (2018).
  23. P. A. Erdman, V. Cavina, R. Fazio, F. Taddei, and V. Giovannetti, Maximum power and corresponding efficiency for two-level heat engines and refrigerators: Optimality of fast cycles, New J. Phys. 21, 103049 (2019).
  24. P. Menczel, T. Pyhäranta, C. Flindt, and K. Brandner, Two-stroke optimization scheme for mesoscopic refrigerators, Phys. Rev. B 99, 224306 (2019).
  25. J.-F. Chen, C.-P. Sun, and H. Dong, Boosting the performance of quantum otto heat engines, Phys. Rev. E 100, 032144 (2019).
  26. P. Abiuso and M. Perarnau-Llobet, Optimal Cycles for Low-Dissipation Heat Engines, Phys. Rev. Lett. 124, 110606 (2020).
  27. V. Cavina, P. A. Erdman, P. Abiuso, L. Tolomeo, and V. Giovannetti, Maximum-power heat engines and refrigerators in the fast-driving regime, Phys. Rev. A 104, 032226 (2021).
  28. R. A. Vargas-Hernández, R. T. Q. Chen, K. A. Jung, and P. Brumer, Fully differentiable optimization protocols for non-equilibrium steady states, New J. Phys. 23, 123006 (2021).
  29. P. Terrén Alonso, P. Abiuso, M. Perarnau-Llobet, and L. Arrachea, Geometric optimization of nonequilibrium adiabatic thermal machines and implementation in a qubit system, PRX Quantum 3, 010326 (2022).
  30. Z. Ye, F. Cerisola, P. Abiuso, J. Anders, M. Perarnau-Llobet, and V. Holubec, Optimal finite-time heat engines under constrained control Phys. Rev. Res. 4, 043130 (2022).
  31. V. Holubec, An exactly solvable model of a stochastic heat engine: Optimization of power, power fluctuations and efficiency, J. Stat. Mech.: Theory Exp. (2014) P05022.
  32. H. J. D. Miller and M. Mehboudi, Geometry of Work Fluctuations Versus Efficiency in Microscopic Thermal Machines, Phys. Rev. Lett. 125, 260602 (2020).
  33. T. Denzler and E. Lutz, Power fluctuations in a finite-time quantum carnot engine, Phys. Rev. Res. 3, L032041 (2021).
  34. S. Saryal and B. K. Agarwalla, Bounds on fluctuations for finite-time quantum otto cycle, Phys. Rev. E 103, L060103 (2021).
  35. M. Mehboudi and H. J. D. Miller, Thermodynamic length and work optimisation for gaussian quantum states, Phys. Rev. A 105, 062434 (2022).
  36. A. C. Barato and U. Seifert, Thermodynamic Uncertainty Relation for Biomolecular Processes, Phys. Rev. Lett. 114, 158101 (2015).
  37. T. R. Gingrich, J. M. Horowitz, N. Perunov, and J. L. England, Dissipation Bounds All Steady-State Current Fluctuations, Phys. Rev. Lett. 116, 120601 (2016).
  38. P. Pietzonka and U. Seifert, Universal Trade-Off between Power, Efficiency, and Constancy in Steady-State Heat Engines, Phys. Rev. Lett. 120, 190602 (2018).
  39. G. Guarnieri, G. T. Landi, S. R. Clark, and J. Goold, Thermodynamics of precision in quantum nonequilibrium steady states, Phys. Rev. Res. 1, 033021 (2019).
  40. A. M. Timpanaro, G. Guarnieri, J. Goold, and G. T. Landi, Thermodynamic Uncertainty Relations from Exchange Fluctuation Theorems, Phys. Rev. Lett. 123, 090604 (2019).
  41. G. Falasco, M. Esposito, and J.-C. Delvenne, Unifying thermodynamic uncertainty relations, New J. Phys. 22, 053046 (2020).
  42. H. M. Friedman, B. K. Agarwalla, O. Shein-Lumbroso, O. Tal, and D. Segal, Thermodynamic uncertainty relation in atomic-scale quantum conductors, Phys. Rev. B 101, 195423 (2020).
  43. J. M. Horowitz and T. R. Gingrich, Thermodynamic uncertainty relations constrain non-equilibrium fluctuations, Nat. Phys. 16, 15 (2020).
  44. S. Saryal, M. Gerry, I. Khait, D. Segal, and B. K. Agarwalla, Universal Bounds on Fluctuations in Continuous Thermal Machines, Phys. Rev. Lett. 127, 190603 (2021).
  45. B. K. Agarwalla and D. Segal, Assessing the validity of the thermodynamic uncertainty relation in quantum systems, Phys. Rev. B 98, 155438 (2018).
  46. K. Brandner, T. Hanazato, and K. Saito, Thermodynamic Bounds on Precision in Ballistic Multiterminal Transport, Phys. Rev. Lett. 120, 090601 (2018).
  47. K. Ptaszyński, Coherence-enhanced constancy of a quantum thermoelectric generator, Phys. Rev. B 98, 085425 (2018).
  48. F. Carollo, R. L. Jack, and J. P. Garrahan, Unraveling the Large Deviation Statistics of Markovian Open Quantum Systems, Phys. Rev. Lett. 122, 130605 (2019).
  49. J. Liu and D. Segal, Thermodynamic uncertainty relation in quantum thermoelectric junctions, Phys. Rev. E 99, 062141 (2019).
  50. A. Arash Sand Kalaee, A. Wacker, and P. P. Potts, Violating the thermodynamic uncertainty relation in the three-level maser, Phys. Rev. E 104, L012103 (2021).
  51. A. Rignon-Bret, G. Guarnieri, J. Goold, and M. T. Mitchison, Thermodynamics of precision in quantum nanomachines, Phys. Rev. E 103, 012133 (2021).
  52. T. Ehrlich and G. Schaller, Broadband frequency filters with quantum dot chains, Phys. Rev. B 104, 045424 (2021).
  53. M. Gerry and D. Segal, Absence and recovery of cost-precision tradeoff relations in quantum transport, Phys. Rev. B 105, 155401 (2022).
  54. A. C. Barato and U. Seifert, Cost and Precision of Brownian Clocks, Phys. Rev. X 6, 041053 (2016).
  55. K. Proesmans and C. V. den Broeck, Discrete-time thermodynamic uncertainty relation, Europhys. Lett. 119, 20001 (2017).
  56. V. Holubec and A. Ryabov, Cycling Tames Power Fluctuations near Optimum Efficiency, Phys. Rev. Lett. 121, 120601 (2018).
  57. L. M. Cangemi, V. Cataudella, G. Benenti, M. Sassetti, and G. De Filippis, Violation of thermodynamics uncertainty relations in a periodically driven work-to-work converter from weak to strong dissipation, Phys. Rev. B 102, 165418 (2020).
  58. P. Menczel, E. Loisa, K. Brandner, and C. Flindt, Thermodynamic uncertainty relations for coherently driven open quantum systems, J. Phys. A: Math. Theor. 54, 314002 (2021).
  59. J. Lu, Z. Wang, J. Peng, C. Wang, J.-H. Jiang, and J. Ren, Geometric thermodynamic uncertainty relation in a periodically driven thermoelectric heat engine, Phys. Rev. B 105, 115428 (2022).
  60. T. Koyuk, U. Seifert, and P. Pietzonka, A generalization of the thermodynamic uncertainty relation to periodically driven systems, J. Phys. A: Math. Theor. 52, 02LT02 (2019).
  61. T. Koyuk and U. Seifert, Operationally Accessible Bounds on Fluctuations and Entropy Production in Periodically Driven Systems, Phys. Rev. Lett. 122, 230601 (2019).
  62. T. Koyuk and U. Seifert, Thermodynamic Uncertainty Relation for Time-Dependent Driving, Phys. Rev. Lett. 125, 260604 (2020).
  63. T. Van Vu and Y. Hasegawa, Thermodynamic uncertainty relations under arbitrary control protocols, Phys. Rev. Res. 2, 013060 (2020).
  64. E. Potanina, C. Flindt, M. Moskalets, and K. Brandner, Thermodynamic Bounds on Coherent Transport in Periodically Driven Conductors, Phys. Rev. X 11, 021013 (2021).
  65. Y. Hasegawa, Irreversibility, Loschmidt Echo, and Thermodynamic Uncertainty Relation, Phys. Rev. Lett. 127, 240602 (2021).
  66. H. J. D. Miller, M. H. Mohammady, M. Perarnau-Llobet, and G. Guarnieri, Thermodynamic Uncertainty Relation in Slowly Driven Quantum Heat Engines, Phys. Rev. Lett. 126, 210603 (2021).
  67. H. J. D. Miller, M. H. Mohammady, M. Perarnau-Llobet, and G. Guarnieri, Joint statistics of work and entropy production along quantum trajectories, Phys. Rev. E 103, 052138 (2021).
  68. V. Gorini, A. Kossakowski, and E. C. G. Sudarshan, Completely positive dynamical semigroups of N-level systems, J. Math. Phys. 17, 821 (1976).
  69. G. Lindblad, On the generators of quantum dynamical semigroups, Commun. Math. Phys. 48, 119 (1976).
  70. H. Breuer and F. Petruccione, The Theory of Open Quantum systems (Oxford University Press, New York, 2002).
  71. M. Yamaguchi, T. Yuge, and T. Ogawa, Markovian quantum master equation beyond adiabatic regime, Phys. Rev. E 95, 012136 (2017).
  72. D. E. Kirk, Optimal Control Theory: an Introduction (Dover, New York, 2004).
  73. R. S. Sutton and A. G. Barto, Reinforcement Larning: An Introduction (MIT Press, Cambridge, MA, 2018).
  74. B. Sothmann, R. Sánchez, and A. N. Jordan, Thermoelectric energy harvesting with quantum dots, Nanotechnology 26, 032001 (2015).
  75. J. P. Pekola, B. Karimi, G. Thomas, and D. V. Averin, Supremacy of incoherent sudden cycles, Phys. Rev. B 100, 085405 (2019).
  76. L. M. Cangemi, M. Carrega, A. De Candia, V. Cataudella, G. De Filippis, M. Sassetti, and G. Benenti, Optimal energy conversion through antiadiabatic driving breaking time-reversal symmetry, Phys. Rev. Res. 3, 013237 (2021).
  77. P. Salamon and R. S. Berry, Thermodynamic Length and Dissipated Availability, Phys. Rev. Lett. 51, 1127 (1983).
  78. J. Wang, J. He, and X. He, Performance analysis of a two-state quantum heat engine working with a single-mode radiation field in a cavity, Phys. Rev. E 84, 041127 (2011).
  79. J. E. Avron, M. Fraas, G. M. Graf, and P. Grech, Adiabatic theorems for generators of contracting evolutions, Commun. Math. Phys. 314, 163 (2012).
  80. M. F. Ludovico, F. Battista, F. von Oppen, and L. Arrachea, Adiabatic response and quantum thermoelectrics for ac-driven quantum systems, Phys. Rev. B 93, 075136 (2016).
  81. V. Cavina, A. Mari, and V. Giovannetti, Slow Dynamics and Thermodynamics of Open Quantum Systems, Phys. Rev. Lett. 119, 050601 (2017).
  82. H. J. D. Miller, M. Scandi, J. Anders, and M. Perarnau-Llobet, Work Fluctuations in Slow Processes: Quantum Signatures and Optimal Control, Phys. Rev. Lett. 123, 230603 (2019).
  83. M. Scandi and M. Perarnau-Llobet, Thermodynamic length in open quantum systems, Quantum 3, 197 (2019).
  84. K. Brandner and K. Saito, Thermodynamic Geometry of Microscopic Heat Engines, Phys. Rev. Lett. 124, 040602 (2020).
  85. B. Bhandari, P. T. Alonso, F. Taddei, F. von Oppen, R. Fazio, and L. Arrachea, Geometric properties of adiabatic quantum thermal machines, Phys. Rev. B 102, 155407 (2020).
  86. P. Abiuso, H. J. D. Miller, M. Perarnau-Llobet, and M. Scandi, Geometric optimisation of quantum thermodynamic processes, Entropy 22, 1076 (2020).
  87. A. G. Frim and M. R. DeWeese, Geometric Bound on the Efficiency of Irreversible Thermodynamic Cycles, Phys. Rev. Lett. 128, 230601 (2022).
  88. J. Eglinton and K. Brandner, Geometric bounds on the power of adiabatic thermal machines, Phys. Rev. E 105, L052102 (2022).
  89. K. Brandner, K. Saito, and U. Seifert, Thermodynamics of Micro- and Nano-Systems Driven by Periodic Temperature Variations, Phys. Rev. X 5, 031019 (2015).
  90. K. Brandner and U. Seifert, Periodic thermodynamics of open quantum systems, Phys. Rev. E 93, 062134 (2016).
  91. O. Movilla Miangolarra, R. Fu, A. Taghvaei, Y. Chen, and T. T. Georgiou, Underdamped stochastic thermodynamic engines in contact with a heat bath with arbitrary temperature profile, Phys. Rev. E 103, 062103 (2021).
  92. O. Movilla Miangolarra, A. Taghvaei, R. Fu, Y. Chen, and T. T. Georgiou, Energy harvesting from anisotropic fluctuations, Phys. Rev. E 104, 044101 (2021).
  93. See Supplemental Material at http://link.aps.org/supplemental/10.1103/PhysRevResearch.5.L022017 for details on the the general framework, the reinforcement learning method, and the fast and slow driving regimes.
  94. R. Alicki, The quantum open system as a model of the heat engine, J. Phys. A 12, L103 (1979).
  95. L. F. Seoane and R. Solé, Multiobjective optimization and phase transitions, in Proceedings of ECCS 2014, edited by S. Battiston, F. De Pellegrini, G. Caldarelli, and E. Merelli (Springer, Cham, 2016), pp. 259-270.
  96. A. P. Solon and J. M. Horowitz, Phase Transition in Protocols Minimizing Work Fluctuations, Phys. Rev. Lett. 120, 180605 (2018).
  97. T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine, Soft actor-critic algorithms and applications, arXiv:1812.05905.
  98. O. Delalleau, M. Peter, E. Alonso, A. Logut, Discrete and continuous action representation for practical RL in video games, arXiv:1912.11077.
  99. T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, International Conference on Machine Learning, Vol. 80 (PMLR, Birmingham, UK, 2018), p. 1861.
  100. P. A. Erdman and F. Noé, Identifying optimal cycles in quantum thermal machines with reinforcement-learning, npj Quantum Inf 8, 1 (2022).
  101. P. A. Erdman and F. Noé, Driving black-box quantum thermal machines with optimal power/efficiency trade-offs using reinforcement learning, arXiv:2204.04785.
  102. P. Sgroi, G. M. Palma, and M. Paternostro, Reinforcement Learning Approach to Nonequilibrium Quantum Thermodynamics, Phys. Rev. Lett. 126, 020601 (2021).
  103. I. Khait, J. Carrasquilla, and D. Segal, Optimal control of quantum thermal machines using machine learning, Phys. Rev. Res. 4, L012029 (2022).
  104. Y. Ashida and T. Sagawa, Learning the best nanoscale heat engines through evolving network topology, Commun. Phys. 4, 45 (2021).
  105. M. Bukov, A. G. R. Day, D. Sels, P. Weinberg, A. Polkovnikov, and P. Mehta, Reinforcement Learning in Different Phases of Quantum Control, Phys. Rev. X 8, 031086 (2018).
  106. T. Fösel, P. Tighineanu, T. Weiss, and F. Marquardt, Reinforcement Learning with Neural Networks for Quantum Feedback, Phys. Rev. X 8, 031084 (2018).
  107. Z. An and D. Zhou, Deep reinforcement learning for quantum gate control, Europhys. Lett. 126, 60002 (2019).
  108. M. Y. Niu, S. Boixo, V. N. Smelyanskiy, and H. Neven, Universal quantum control through deep reinforcement learning, npj Quantum Inf 5, 33 (2019).
  109. X.-M. Zhang, Z. Wei, R. Asad, X.-C. Yang, and X. Wang, When does reinforcement learning stand out in quantum control? A comparative study on state preparation, npj Quantum Inf 5, 85 (2019).
  110. M. Dalgaard, F. Motzoi, J. J. Sørensen, and J. Sherson, Global optimization of quantum dynamics with alphazero deep exploration, npj Quantum Inf 6, 6 (2020).
  111. J. Mackeprang, D. B. R. Dasari, and J. Wrachtrup, A reinforcement learning approach for quantum state engineering, Quantum Mach. Intell. 2, 5 (2020).
  112. F. Schäfer, M. Kloc, C. Bruder, and N. Lörch, A differentiable programming method for quantum control, Mach. Learn.: Sci. Technol. 1, 035009 (2020).
  113. R. Sweke, M. S. Kesselring, E. P. L. van Nieuwenburg, and J. Eisert, Reinforcement learning decoders for fault-tolerant quantum computation, Mach. Learn.: Sci. Technol. 2, 025005 (2021).
  114. F. Schäfer, P. Sekatski, M. Koppenhöfer, C. Bruder, and M. Kloc, Control of stochastic quantum dynamics by differentiable programming, Mach. Learn.: Sci. Technol. 2, 035004 (2021).
  115. R. Porotti, A. Essig, B. Huard, and F. Marquardt, Deep reinforcement learning for quantum state preparation with weak nonlinear measurements, Quantum 6, 747 (2022).
  116. F. Metz and M. Bukov, Self-correcting quantum many-body control using reinforcement learning with tensor networks, arXiv:2201.11790.
  117. L. Coopmans, S. Campbell, G. De Chiara, and A. Kiely, Optimal control in disordered quantum systems, Phys. Rev. Res. 4, 043138 (2022).

Outline

Information

Phys. Rev. Research 5, L022017– Published 27 April, 2023
  • Received 2 August 2022
  • Revised 4 December 2022
  • Accepted 15 March 2023
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation