- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Accelerators and Beams

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Real-time reinforcement learning on coupling correction

Yihao Gong, Shunqiang Tian*, and Linglong Mao

Xinzhong Liu, Xuan Shouzhi, and Liyuan Tan

  • *Contact author: tiansq@sari.ac.cn
PDF

Phys. Rev. Accel. Beams 29, 104002 – Published 20 October, 2026

DOI: https://doi.org/10.1103/49ns-fryv

Abstract

The stability of vertical emittance is a critical performance metric for modern high-brightness synchrotron light sources. This paper presents a real-time, reinforcement learning (RL)-based correction framework based on deep reinforcement learning by employing dueling deep Q-network with a discrete action space to dynamically adjust skew quadrupole magnets. To ensure operational safety and accelerate convergence, we introduce a curriculum learning strategy and leverage a high-fidelity simulation of the Shanghai Synchrotron Radiation Facility for systematic hyperparameter optimization. Experimental results demonstrate that the agent, trained in situ on the real machine, successfully achieves stabilization of the beam-size readback at the submicrometer fluctuation level and exhibits rapid adaptability to target changes. This work validates the feasibility of using RL for robust, online accelerator optimization during user beam delivery.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (38)

  1. P. Raimondi, N. Carmignani, L. R. Carver et al., Commissioning of the hybrid multibend achromat lattice at the European synchrotron radiation facility, Phys. Rev. Accel. Beams 24, 110701 (2021).
  2. P. F. Tavares, E. Al-Dmour, Åke Andersson, F. Cullinan, B. N. Jensen, D. Olsson, D. K. Olsson, M. Sjöström, H. Tarawneh, S. Thorin, and A. Vorozhtsov, Commissioning and first-year operational results of the MAX IV 3 GeV ring, J. Synchrotron Radiat. 25, 1291 (2018).
  3. C. Steier, A. Allézy, A. Anders et al., Status of the conceptual design of ALS-U, in Proc. 9th Int. Particle Accelerator Conf. (IPAC’18), Vancouver, BC, Canada (JACoW Publishing, Geneva, Switzerland, 2018), pp. 4136–4139.
  4. Y. Jiao, G. Xu, X. H. Cui et al., The HEPS project, J. Synchrotron Radiat. 25, 1611 (2018).
  5. M. Aiba, M. Böge, and A. Streun, Ultra-low vertical emittance at SLS, in Proc. 3rd Int. Particle Accelerator Conf. (IPAC’12), New Orleans, LA, USA (JACoW Publishing, Geneva, Switzerland, 2012), pp. 843–845.
  6. X. Liu, S. Tian, L. Tan, Y. Gong, S. Xuan, and Z. Zhao, Applications of vertical damping wigglers in an x-ray diffraction limited storage ring, Nucl. Instrum. Methods Phys. Res., Sect. A 1056, 168653 (2023).
  7. A. Franchi, L. Farvacque, J. Chavanne, F. Ewald, B. Nash, K. Scheidt, and R. Tomás, Vertical emittance reduction and preservation in electron storage rings via resonance driving terms correction, Phys. Rev. ST Accel. Beams 14, 034002 (2011).
  8. X.-Z. Liu, S.-Q. Tian, X. Wu, M. Wang, Z.-T. Zhao, and B.-Y. Feng, Feedforward compensation of the insertion devices effects in the SSRF storage ring, Nucl. Sci. Tech. 33, 70 (2022).
  9. Y. Hidaka, Y. Li, T. Shaftan, T. Tanabe, Y. Tian, and G. Wang, Correction of ID-induced transverse linear coupling at NSLS-II, in 9th International Particle Accelerator Conference (JACoW Publishing, Geneva, Switzerland, 2018).
  10. A. L. Edelen, S. G. Biedron, B. E. Chase, D. Edstrom, S. V. Milton, and P. Stabile, Neural networks for modeling and control of particle accelerators, IEEE Trans. Nucl. Sci. 63, 878 (2016).
  11. S. C. Leemann, S. Liu, A. Hexemer, M. A. Marcus, C. N. Melton, H. Nishimura, and C. Sun, Demonstration of machine learning-based model-independent stabilization of source properties in synchrotron light sources, Phys. Rev. Lett. 123, 194801 (2019).
  12. T. Hellert, T. Ford, S. C. Leemann, H. Nishimura, M. Venturini, and A. Pollastro, Application of deep learning methods for beam size control during user operation at the advanced light source, Phys. Rev. Accel. Beams 27, 074602 (2024).
  13. J. Duris, D. Kennedy, A. Hanuka, J. Shtalenkova, A. Edelen, P. Baxevanis, A. Egger, T. Cope, M. McIntire, S. Ermon, and D. Ratner, Bayesian optimization of a free-electron laser, Phys. Rev. Lett. 124, 124801 (2020).
  14. R. Roussel, A. Hanuka, and A. Edelen, Multiobjective Bayesian optimization for online accelerator tuning, Phys. Rev. Accel. Beams 24, 062801 (2021).
  15. R. Roussel et al., Bayesian optimization algorithms for accelerator physics, Phys. Rev. Accel. Beams 27, 084801 (2024).
  16. J. Kaiser, C. Xu, A. Eichler, A. S. Garcia, O. Stein, E. Bründermann, W. Kuropka, H. Dinter, F. Mayet, T. Vinatier, F. Burkart, and H. Schlarb, Reinforcement learning-trained optimisers and Bayesian optimisation for online particle accelerator tuning, Sci. Rep. 14, 15733 (2024).
  17. V. Kain, S. Hirlander, B. Goddard, F. M. Velotti, G. Z. Della Porta, N. Bruchon, and G. Valentino, Sample-efficient reinforcement learning for CERN accelerator control, Phys. Rev. Accel. Beams 23, 124801 (2020).
  18. X. Chen, Y. Jia, X. Qi, Z. Wang, and Y. He, Orbit correction based on improved reinforcement learning algorithm, Phys. Rev. Accel. Beams 26, 044601 (2023).
  19. L. Grech, G. Valentino, D. Alves, and S. Hirlaender, Application of reinforcement learning in the LHC tune feedback, Front. Phys. 10, 929064 (2022).
  20. F. H. O’Shea, N. Bruchon, and G. Gaio, Policy gradient methods for free-electron laser and terahertz source optimization and stabilization at the fermi free-electron laser at Elettra, Phys. Rev. Accel. Beams 23, 122802 (2020).
  21. W. Wang, M. Caselle, T. Boltz, E. Blomley, M. Brosi, T. Dritschler, A. Ebersoldt, A. Kopmann, A. Santamaria Garcia, P. Schreiber, E. Bründermann, M. Weber, A.-S. Müller, and Y. Fang, Accelerated deep reinforcement learning for fast feedback of beam dynamics at KARA, IEEE Trans. Nucl. Sci. 68, 1794 (2021).
  22. A. Scheinker, A. Edelen, D. Bohler, C. Emma, and A. Lutman, Demonstration of model-independent control of the longitudinal phase space of electron beams in the linac-coherent light source with femtosecond resolution, Phys. Rev. Lett. 121, 044801 (2018).
  23. A. Scheinker, S. Hirlaender, F. M. Velotti, S. Gessner, G. Z. Della Porta, V. Kain, B. Goddard, and R. Ramjiawan, Online multi-objective particle accelerator optimization of the awake electron beam line for simultaneous emittance and orbit control, AIP Adv. 10, 055320 (2020).
  24. V. Mnih, K. Kavukcuoglu, D. Silver et al., Human-level control through deep reinforcement learning, Nature (London) 518, 529 (2015).
  25. Z. Wang, T. Schaul, M. Hessel, H. van Hasselt, M. Lanctot, and N. de Freitas, Dueling network architectures for deep reinforcement learning, in Proceedings of the 33rd International Conference on Machine Learning (ICML) (PMLR, New York, USA, 2016), Vol. 48, pp. 1995–2003.
  26. H. Xu and Z. Zhao, Current status and progresses of SSRF project, Nucl. Sci. Tech. 19, 1 (2008).
  27. A. Wolski, Low-emittance storage rings, CERN, Tech. Rep. CERN-2014-009, 2014.
  28. C. L. Li, S. Q. Tian, and W. Z. Zhang, Study on the vertical emittance for the SSRF phase II project, Nucl. Sci. Tech. 39, 050102 (2016).
  29. J. Safranek, Experimental determination of storage ring optics using orbit response measurements, Nucl. Instrum. Methods Phys. Res., Sect. A 388, 27 (1997).
  30. D. A. Edwards and L. C. Teng, Parametrization of linear coupled motion in periodic systems, IEEE Trans. Nucl. Sci. 20, 885 (1973).
  31. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed. (MIT Press, Cambridge, MA, USA, 2018).
  32. A. Terebilo, Accelerator toolbox for MATLAB, SLAC National Accelerator Laboratory, Tech. Rep. SLAC-PUB-8732, 2001.
  33. X. Wu, S. Tian, X. Liu, W. Zhang, and Z. Zhao, Design and commissioning of the new SSRF storage ring lattice with asymmetric optics, Nucl. Instrum. Methods Phys. Res., Sect. A 1025, 166098 (2022).
  34. G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, Openai gym, arXiv:1606.01540.
  35. Y. Bengio, J. Louradour, R. Collobert, and J. Weston, Curriculum learning, in Proceedings of the 25th International Conference on Machine Learning (ICML) (Association for Computing Machinery, New York, NY, USA, 2009), pp. 41–48.
  36. F. M. Velotti, B. Goddard, V. Kain, R. Ramjiawan, G. Zevi Della Porta, and S. Hirlaender, Towards automatic setup of 18 MeV electron beamline using machine learning, Mach. Learn. 4, 025016 (2023).
  37. A. Awal, J. Hetzel, R. Gebel, and J. Pretz, Injection optimization at particle accelerators via reinforcement learning: From simulation to real-world application, Phys. Rev. Accel. Beams 28, 034601 (2025).
  38. T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, Optuna: A next-generation hyperparameter optimization framework, in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’19) (Association for Computing Machinery, New York, NY, USA, 2019), pp. 2623–2631.

Outline

Information

Phys. Rev. Accel. Beams 29, 104002– Published 20 October, 2026
  • Received 11 January 2026
  • Accepted 4 September 2026
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation