- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Accelerators and Beams

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Batch spacing optimization by reinforcement learning

Matthias Remta and Francesco Velotti

Sharwin Rezagholi

  • CERN, Geneva 1211, Switzerland

PDF

Phys. Rev. Accel. Beams 28, 094603 – Published 25 September, 2025

DOI: https://doi.org/10.1103/g9wr-197z

Abstract

Beams designated for the LHC are injected into the SPS in multiple batches. Given the tight spacing of 200 ns between these batches, the injection kickers have to be precisely synchronized with the injected beam to minimize injection oscillations. Due to machine drift, the optimal settings for the kickers vary. This paper presents an active controller trained by reinforcement learning that counteracts the machine drifts by adjusting the settings. The agent was exclusively trained in a simulation environment and directly transferred to the accelerator. Although its results are slightly worse than those obtained by an explicit numerical optimizer, the BOBYQA algorithm, the agent attains these results much faster since it requires far less computation.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (46)

  1. W. Bartmann, M. Barnes, J. Boyd, E. Carlier, A. Chmielinska, B. Goddard, G. Kotzian, C. Schwick, L. Stoel, D. Valuch, F. Velotti, V. Vlachodimitropoulos, and C. Wiesner, Impact of LHC and SPS injection kicker rise times on LHC filling schemes and luminosity reach, in Proceedings of the International Particle Accelerator Conference (IPAC-2017) (JACoW, Geneva, Switzerland, 2017), pp. 2043–2046.
  2. E. Waagaard, Optimising the present and designing the future: A novel SPS injection system, Master’s thesis, Uppsala University, Uppsala, Sweden, 2022, https://cds.cern.ch/record/2813209.
  3. M. Powell, The BOBYQA algorithm for bound constrained optimization without derivatives, Technical Report No. 2009/NA06, University of Cambridge, Department of Applied Mathematics and Theoretical Physics, 2009.
  4. I. Bogunovic, J. Scarlett, and V. Cevher, Time-varying Gaussian process bandit optimization, in Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, Volume 51 of Proceedings of Machine Learning Research, Cadiz, Spain, edited by A. Gretton and C. C. Robert (2016), pp. 314–323, https://proceedings.mlr.press/v51/bogunovic16.html.
  5. N. Kuklev et al., Robust adaptive Bayesian optimization, in Proceedings of the 14th International Particle Accelerator Conference, IPAC-2023 (JACoW, Geneva, Switzerland, 2023), pp. 4428–4431, 10.18429/JACoW-IPAC2023-THPL007.
  6. N. Kuklev, M. Borland, G. Fystro, H. Shang, and Y. Sun, Online accelerator tuning with adaptive Bayesian optimization, in Proceedings of the 5th International Particle Accelerator Conference (NAPAC’22), Number 5 in International Particle Accelerator Conference (JACoW, Geneva, Switzerland, 2022), Vol. 10, pp. 842–845, 10.18429/JACoW-NAPAC2022-THXD4.
  7. R. Roussel et al., Bayesian optimization algorithms for accelerator physics, Phys. Rev. Accel. Beams 27, 084801 (2024).
  8. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms, arXiv:1707.06347.
  9. M. Remta, mkp-delays-rl, GitLab, 2024, https://gitlab.cern.ch/mremta/mkp-delays-rl [accessed January 8, 2025].
  10. L. Grech, G. Valentino, D. Alves, and S. Hirlaender, Application of reinforcement learning in the LHC tune feedback, Front. Phys. 10, 929064 (2022).
  11. V. Kain, S. Hirlander, B. Goddard, F. M. Velotti, G. Z. Della Porta, N. Bruchon, and G. Valentino, Sample-efficient reinforcement learning for CERN accelerator control, Phys. Rev. Accel. Beams 23, 124801 (2020).
  12. M. Schenk, E. F. Combarro, M. Grossi, V. Kain, K. S. B. Li, M.-M. Popa, and S. Vallecorsa, Hybrid actor-critic algorithm for quantum reinforcement learning at CERN beam lines, arXiv:2209.11044.
  13. F. Velotti, B. Goddard, V. Kain, R. Ramjiawan, G. Porta, and S. Hirlander, Towards automatic setup of 18 MeV electron beamline using machine learning, Mach. Learn. 4, 025016 (2023).
  14. D. Wang, H. Bagri, C. Macdonald, S. Kiy, P. Jung, O. Shelbaya, T. Planche, W. Fedorko, R. Baartman, and O. Kester, Accelerator tuning with deep reinforcement learning, in Proceedings of the 4th Workshop on Machine Learning and the Physical Sciences (2021), https://ml4physicalsciences.github.io/2021/files/NeurIPS_ML4PS_2021_125.pdf.
  15. X. Chen, Y. Jia, X. Qi, Z. Wang, and Y. He, Orbit correction based on improved reinforcement learning algorithm, Phys. Rev. Accel. Beams 26, 044601 (2023).
  16. N. Madysa, R. Alemany-Fernández, N. Biancacci, B. Goddard, V. Kain, and F. Velotti, Automated intensity optimisation using reinforcement learning at LEIR, in Proceedings of the13th International Particle Accelerator Conference, IPAC-2022, Bangkok, Thailand (JACoW, Geneva, Switzerland, 2022), pp. 941–944.
  17. M. Schram, K. Rajput, N. S. K. S., P. Li, J. St. John, and H. Sharma, Uncertainty aware machine-learning-based surrogate models for particle accelerators: Study at the Fermilab booster accelerator complex, Phys. Rev. Accel. Beams 26, 044602 (2023).
  18. J. Kaiser, O. Stein, and A. Eichler, Learning-based optimisation of particle accelerators under partial observability without real-world training, in Proceedings of the 39th International Conference on Machine Learning (2022), pp. 10575–10585, https://proceedings.mlr.press/v162/kaiser22a.html.
  19. J. Kaiser, C. Xu, A. Eichler, A. S. Garcia, O. Stein, E. Bründermann, W. Kuropka, H. Dinter, F. Mayet, T. Vinatier, F. Burkart, and H. Schlarb, Learning to do or learning while doing: Reinforcement learning and Bayesian optimisation for online continuous tuning, arXiv:2306.03739.
  20. S. Gu, T. Lillicrap, I. Sutskever, and S. Levine, Continuous deep Q-learning with model-based acceleration, in Proceedings of the 33rd International Conference on Machine Learning (2016), pp. 2829–2838, https://proceedings.mlr.press/v48/gu16.html.
  21. S. Fujimoto, H. van Hoof, and D. Meger, Addressing function approximation error in actor-critic methods, in Proceedings of the 35th International Conference on Machine Learning (2018), pp. 1587–1596, https://proceedings.mlr.press/v80/fujimoto18a.html.
  22. S. Hirlaender, S. Pochaba, L. Lamminger, A. Santamaria Garcia, C. Xu, J. Kaiser, A. Eichler, and V. Kain, Deep meta reinforcement learning for rapid adaptation in linear Markov decision processes: Applications to CERN’s AWAKE project, in Volume 1458 of Advances in Intelligent Systems and Computing (Springer Nature, Switzerland, 2024), 10.1007/978-3-031-65993-5_21.
  23. C. Finn, P. Abbeel, and S. Levine, Model-agnostic meta-learning for fast adaptation of deep networks, in Proceedings of the 34th International Conference on Machine Learning, Volume 70 of Proceedings of Machine Learning Research, edited by D. Precup and Y. W. Teh (2017), pp. 1126–1135, https://proceedings.mlr.press/v70/finn17a.html.
  24. D. Meier, L. V. Ramirez, J. Völker, J. Viefhaus, B. Sick, and G. Hartmann, Optimizing a superconducting radio-frequency gun using deep reinforcement learning, Phys. Rev. Accel. Beams 25, 104604 (2022).
  25. J. St. John, C. Herwig, D. Kafkes, J. Mitrevski, W. A. Pellico, G. N. Perdue, A. Quintero-Parra, B. A. Schupbach, K. Seiya, N. Tran, M. Schram, J. M. Duarte, Y. Huang, and R. Keller, Real-time artificial intelligence for accelerator control: A study at the Fermilab booster, Phys. Rev. Accel. Beams 24, 104601 (2021).
  26. H. Hasselt, Double q-learning, in Advances in Neural Information Processing Systems, edited by J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta (Curran Associates, Inc., Red Hook, NY, 2010), Vol. 23, https://proceedings.neurips.cc/paper_files/paper/2010/file/091d584fced301b442654dd8c23b3fc9-Paper.pdf.
  27. N. Bruchon, G. Fenu, G. Gaio, M. Lonza, F. H. O’Shea, F. A. Pellegrino, and E. Salvato, Basic reinforcement learning techniques to control the intensity of a seeded free-electron laser, Electronics 9, 781 (2020).
  28. V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, Playing Atari with deep reinforcement learning, arXiv:1312.5602.
  29. R. J. Williams, Simple statistical gradient-following algorithms for connectionist reinforcement learning, Mach. Learn. 8, 229 (1992).
  30. S. M. Kakade, A natural policy gradient, in Proceedings of the Advances in Neural Information Processing Systems 14, NIPS 2001 (2001), https://proceedings.neurips.cc/paper_files/paper/2001/file/4b86abe48d358ecf194c56c69108433e-Paper.pdf.
  31. N. Bruchon, G. Fenu, G. Gaio, S. Hirlaender, M. Lonza, F. A. Pellegrino, and E. Salvato, An online iterative linear quadratic approach for a satisfactory working point attainment at Fermi, Information 12, 262 (2021).
  32. F. H. O’Shea, N. Bruchon, and G. Gaio, Policy gradient methods for free-electron laser and terahertz source optimization and stabilization at the Fermi free-electron laser at Elettra, Phys. Rev. Accel. Beams 23, 122802 (2020).
  33. S. Hochreiter and J. Schmidhuber, Long short-term memory, Neural Comput. 9, 1735 (1997).
  34. E. Waagaard, M. Barnes, W. Bartmann, V. Bencini, J. Borburgh, L. Ducimetière, T. Kramer, T. Stadlbauer, P. Trubacova, and F. Velotti, Design of a new CERN SPS injection system via numerical optimisation, in Proceedings of the 14th International Particle Accelerator Conference (IPAC-2023) (JACoW, Geneva, Switzerland, 2023), pp. 267–270.
  35. J. Uythoven, The new SPS injection channel, in Proceedings of the 9th LEP Performance Workshop (1999), pp. 120–125, https://cds.cern.ch/record/398068.
  36. R. Sutton and A. Barto, Reinforcement Learning: An Introduction, 2nd ed. (MIT Press, Cambridge, MA, 2018).
  37. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms, arXiv:1707.06347.
  38. StableBaselines3, Recurrent PPO, https://sb3-contrib.readthedocs.io/en/master/modules/ppo_recurrent.html (2024) [accessed April 26, 2024].
  39. A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, Stable-baselines3: Reliable reinforcement learning implementations, J. Mach. Learn. Res. 22, 1 (2021), http://jmlr.org/papers/v22/20-1364.html.
  40. StableBaselines3, PPO, https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html (2024) [accessed February 12, 2024].
  41. mad-x project, mad-x: Methodical accelerator design, https://cern.ch/madx (2025) [accessed June 2, 2025].
  42. C. Cartis, J. Fiala, B. Marteau, and L. Roberts, Improving the flexibility and robustness of model-based derivative-free optimization solvers, ACM Trans. Math. Softw. 45, 1 (2019).
  43. C. Cartis, L. Roberts, and O. Sheridan-Methven, Escaping local minima with local derivative-free methods: A numerical investigation, Optimization 71, 2343 (2021).
  44. T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, Optuna: A next-generation hyperparameter optimization framework, in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (2019), pp. 2623–2631, 10.1145/3292500.3330701.
  45. J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl, Algorithms for hyper-parameter optimization, in Proceedings of the 24th International Conference on Neural Information Processing Systems (2011), pp. 2546–2554, https://proceedings.neurips.cc/paper_files/paper/2011/file/86e8f7ab32cfd12577bc2619bc635690-Paper.pdf.
  46. J. Bergstra, D. Yamins, and D. Cox, Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures, in Proceedings of the 30th International Conference on Machine Learning, Atlanta, Georgia (2013), pp. 115–123, https://proceedings.mlr.press/v28/bergstra13.html.

Outline

Information

Phys. Rev. Accel. Beams 28, 094603– Published 25 September, 2025
  • Received 21 January 2025
  • Accepted 3 September 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation