- Physical Review Journals
All JournalsPhysics Magazine

Physical Review D

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Optimizers for stabilizing likelihood-free inference

G. Bruno De Luca1,2,*, Benjamin Nachman3,4,†, Eva Silverstein1,2,‡, and Henry Zheng1,2,§

  • *Contact author: gbdeluca@stanford.edu
  • †Contact author: bpnachman@lbl.gov
  • ‡Contact author: evas@stanford.edu
  • §Contact author: henryzheng@stanford.edu
PDF

Phys. Rev. D 112, 092008 – Published 13 November, 2025

DOI: https://doi.org/10.1103/x88j-mv39

Abstract

A growing number of applications in particle physics and beyond use neural networks as unbinned likelihood ratio estimators applied to real or simulated data. Precision requirements on the inference tasks demand a high-level of stability from these networks, which are affected by the stochastic nature of training. We show how physics concepts can be used to stabilize network training through a physics-inspired optimizer. In particular, the energy conserving descent (ECD) optimization framework uses classical Hamiltonian dynamics on the space of network parameters to reduce the dependence on the initial conditions while also stabilizing the result near the minimum of the loss function. We develop a version of this optimizer known as ECDq=1, which has few free hyperparameters with limited ranges guided by physical reasoning. We apply ECDq=1 to representative likelihood-ratio estimation tasks in particle physics and find on average that it out-performs the widely used Adam optimizer. We expect that ECD will be a useful tool for wide array of data-limited problems, where it is computationally expensive to exhaustively optimize hyperparameters and mitigate fluctuations with ensembling.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (55)

  1. K. Cranmer, J. Brehmer, and G. Louppe, The frontier of simulation-based inference, Proc. Natl. Acad. Sci. U.S.A. 117, 30055 (2020).
  2. Y. Kahn, S. Roy, J. Shelton, and V. Riki, Systematic uncertainties and data complexity in normalizing flows, Neurips Workshop (2024).
  3. V. Andreev et al. (H1 Collaboration), Measurement of lepton-jet correlation in deep-inelastic scattering with the H1 detector using machine learning for unfolding, Phys. Rev. Lett. 128, 132002 (2022).
  4. V. Andreev et al. (H1 Collaboration), Unbinned deep learning jet substructure measurement in high Q2ep collisions at HERA, Phys. Lett. B 844, 138101 (2023).
  5. G. Aad et al. (ATLAS Collaboration), Simultaneous unbinned differential cross-section measurement of twenty-four Z+jets kinematic observables with the ATLAS detector, Phys. Rev. Lett. 133, 261803 (2024).
  6. CMS Collaboration, Measurement of event shapes in minimum bias events from pp collisions at 13 TeV, Technical Reports No. CMS-SMP-23-008; No. CERN-EP-2025-041, CERN, Geneva, 2024.
  7. G. Aad et al. (ATLAS Collaboration), Measurement of off-shell Higgs boson production in the H*→ZZ→4ℓ decay channel using a neural simulation-based inference technique in 13 TeV pp collisions with the ATLAS detector, Rep. Prog. Phys. 88, 057803 (2025).
  8. G. Aad et al. (ATLAS Collaboration), Dijet resonance search with weak supervision using s=13  TeV pp collisions in the ATLAS detector, Phys. Rev. Lett. 125, 131801 (2020).
  9. V. Chekhovsky et al. (CMS Collaboration), Model-agnostic search for dijet resonances with anomalous jet substructure in proton-proton collisions at s=13  TeV, Rep. Prog. Phys. 88, 067802 (2025).
  10. K. Cranmer, J. Pavez, and G. Louppe, Approximating likelihood ratios with calibrated discriminative classifiers, arXiv:1506.02169.
  11. K. Kong, K. T. Matchev, S. Mrenna, and P. Shyamsundar, New machine learning techniques for simulation-based inference: InferoStatic nets, kernel score estimation, and kernel likelihood ratio estimation, arXiv:2210.01680.
  12. J. Brehmer, K. Cranmer, G. Louppe, and J. Pavez, Constraining effective field theories with machine learning, Phys. Rev. Lett. 121, 111801 (2018).
  13. J. Brehmer, K. Cranmer, G. Louppe, and J. Pavez, A guide to constraining effective field theories with machine learning, Phys. Rev. D 98, 052004 (2018).
  14. R. T. D’Agnolo and A. Wulzer, Learning new physics from a machine, Phys. Rev. D 99, 015014 (2019).
  15. B. Nachman and J. Thaler, Learning from many collider events at once, Phys. Rev. D 103, 116013 (2021).
  16. M. Stoye, J. Brehmer, G. Louppe, J. Pavez, and K. Cranmer, Likelihood-free inference with an improved cross-entropy estimator, arXiv:1808.00973.
  17. G. V. Moustakides and K. Basioti, Training neural networks for likelihood/density ratio estimation, arXiv:1911.00405.
  18. N. Jeffrey and B. D. Wandelt, Evidence networks: Simple losses for fast, amortized, neural Bayesian model comparison, Mach. Learn. Sci. Tech. 5, 015008 (2024).
  19. S. Rizvi, M. Pettee, and B. Nachman, Learning likelihood ratios with neural network classifiers, J. High Energy Phys. 02 (2024) 136.
  20. G. B. De Luca and E. Silverstein, Born-infeld (BI) for AI: Energy-conserving descent (ECD) for optimization, in Proceedings of the 39th International Conference on Machine Learning, edited by K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, Vol. 162 of Proceedings of Machine Learning Research (PMLR, Baltimore, Maryland, USA, 2022), pp. 4918–4936.
  21. G. B. De Luca, A. Gatti, and E. Silverstein, Improving energy conserving descent for machine learning: Theory and practice, arXiv:2306.00352.
  22. M. P. Kuchera, R. Ramanujan, J. Z. Taylor, R. R. Strauss, D. Bazin, J. Bradt, and R. Chen, Machine learning methods for track classification in the AT-TPC, Nucl. Instrum. Methods Phys. Res., Sect. A 940, 156 (2019).
  23. A. Chappell and L. H. Whitehead, Application of transfer learning to neutrino interaction classification, Eur. Phys. J. C 82, 1099 (2022).
  24. F. A. Dreyer, R. Grabarczyk, and P. F. Monni, Leveraging universality of jet taggers through transfer learning, Eur. Phys. J. C 82, 564 (2022).
  25. H. Beauchesne, Z.-E. Chen, and C.-W. Chiang, Improving the performance of weak supervision searches using transfer and meta-learning, J. High Energy Phys. 02 (2024) 138.
  26. V. Mikuni and B. Nachman, Solving key challenges in collider physics with foundation models, Phys. Rev. D 111, L051504 (2025).
  27. D. P. Kingma, Adam: A method for stochastic optimization, arXiv:1412.6980.
  28. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Learning representations by back-propagating errors, Nature (London) 323, 533 (1986).
  29. N. Qian, On the momentum term in gradient descent learning algorithms, Neural Netw. 12 1, 145 (1999).
  30. X. Chen et al., Symbolic discovery of optimization algorithms, Adv. Neural Inf. Process. Syst. 36, 49205 (2023).
  31. M. Pagliardini, P. Ablin, and D. Grangier, The ademamix optimizer: Better, faster, older, arXiv:2409.03137.
  32. J. Robnik, G. B. D. Luca, E. Silverstein, and U. Seljak, Microcanonical Hamiltonian Monte Carlo, J. Mach. Learn. Res. 24, 1 (2023), https://jmlr.org/papers/v24/22-1450.html.
  33. J. Robnik and U. Seljak, Fluctuation without dissipation: Microcanonical Langevin Monte Carlo, arXiv:2303.18221.
  34. L. Xiao, Rethinking conventional wisdom in machine learning: From generalization to scaling, arXiv:2409.15156.
  35. D. Decamp et al. (ALEPH Collaboration), ALEPH: A detector for electron-positron annnihilations at LEP, Nucl. Instrum. Methods Phys. Res., Sect. A 294, 121 (1990); 303, 393(E) (1991).
  36. T. Sjostrand, P. Eden, C. Friberg, L. Lonnblad, G. Miu, S. Mrenna, and E. Norrbin, High-energy physics event generation with pythia 6.1, Comput. Phys. Commun. 135, 238 (2001).
  37. R. Brun, F. Bruyant, M. Maire, A. C. McPherson, and P. Zanarini, GEANT3, Report No. CERN-DD-EE-84-1, https://inspirehep.net/literature/252007.
  38. A. Badea, A. Baty, P. Chang, G. M. Innocenti, M. Maggi, C. Mcginn, M. Peters, T.-A. Sheng, J. Thaler, and Y.-J. Lee, Measurements of two-particle correlations in e+e− collisions at 91 GeV with ALEPH archived data, Phys. Rev. Lett. 123, 212002 (2019).
  39. E. Farhi, A QCD test for jets, Phys. Rev. Lett. 39, 1587 (1977).
  40. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, Dropout: A simple way to prevent neural networks from overfitting, J. Mach. Learn. Res. 15, 1929 (2014).
  41. A. Andreassen, P. T. Komiske, E. M. Metodiev, B. Nachman, and J. Thaler, OmniFold: A method to simultaneously unfold all observables, Phys. Rev. Lett. 124, 182001 (2020).
  42. M. Bähr et al., Herwig++ physics and manual, Eur. Phys. J. C 58, 639 (2008).
  43. J. Bellm et al., Herwig 7.0/Herwig++ 3.0 release note, Eur. Phys. J. C 76, 196 (2016).
  44. J. Bellm et al., Herwig 7.1 Release Note, Technical Reports No. CERN-PH-TH-2017-109; No. CERN-TH-2017-109; No. MAN-HEP-2017-08; No. UWTHPH-2017-10; No. IFJPAN-IV-2017-7N; No. IKHEF-2017-026; No. HERWIG-2017-02; No. KA-TP-19-2017; No. MCNET-17-08; No. IPPP-17-40, 2017, https://herwig.hepforge.org/.
  45. T. Sjöstrand, S. Mrenna, and P. Skands, A brief introduction to pythia 8.1, Comput. Phys. Commun. 178, 852 (2008).
  46. T. Sjöstrand, S. Mrenna, and P. Skands, pythia 6.4 physics and manual, J. High Energy Phys. 05 (2006) 026.
  47. T. Sjöstrand, S. Ask, J. R. Christiansen, R. Corke, N. Desai, P. Ilten, S. Mrenna, S. Prestel, C. O. Rasmussen, and P. Z. Skands, An introduction to pythia 8.2, Comput. Phys. Commun. 191, 159 (2015).
  48. J. de Favereau, C. Delaere, P. Demin, A. Giammanco, V. Lemaître, A. Mertens, M. Selvaggi (The DELPHES3 Collaboration), delphes 3: A modular framework for fast simulation of a generic collider experiment, J. High Energy Phys. 02 (2014) 057.
  49. A. J. Larkoski, S. Marzani, G. Soyez, and J. Thaler, Soft drop, J. High Energy Phys. 05 (2014) 146.
  50. J. Thaler and K. Van Tilburg, Identifying boosted objects with N-subjettiness, J. High Energy Phys. 03 (2011) 015.
  51. J. Thaler and K. Van Tilburg, Maximizing boosted top identification by minimizing N-subjettiness, J. High Energy Phys. 02 (2012) 093.
  52. D. Jimenez Rezende and S. Mohamed, Variational inference with normalizing flows, arXiv:1505.05770.
  53. R. T. D’Agnolo and A. Wulzer, Learning new physics from a machine, Phys. Rev. D 99, 015014 (2019).
  54. B. Nachman and J. Thaler, Learning from many collider events at once, Phys. Rev. D 103, 116013 (2021).
  55. https://github.com/zhenghenry/LikelihoodRatioStability/.

Outline

Information

Phys. Rev. D 112, 092008– Published 13 November, 2025
  • Received 6 March 2025
  • Accepted 21 October 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI. Funded by SCOAP3.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation