- Physical Review Journals
All JournalsPhysics Magazine

Physical Review D

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Frequentist uncertainties on neural density ratios with wi fi ensembles

Sean Benevedes* and Jesse Thaler†

  • *Contact author: seanmb@mit.edu
  • †Contact author: jthaler@mit.edu
PDF

Phys. Rev. D 112, 056024 – Published 19 September, 2025

DOI: https://doi.org/10.1103/w28w-x5wh

Abstract

We introduce wifi ensembles as a novel framework to obtain asymptotic frequentist uncertainties on density ratios, with a particular focus on neural ratio estimation in the context of high-energy physics. When the density ratio of interest is a likelihood ratio conditioned on parameters, wifi ensembles can be used to perform simulation-based inference on those parameters. After training the basis functions fi(x), uncertainties on the weights wi can be straightforwardly propagated to the estimated parameters without requiring extraneous bootstraps. To demonstrate this approach, we present an application in quantum chromodynamics at the Large Hadron Collider, using wifi ensembles to estimate the likelihood ratio between generated quark and gluon jets. We use this learned likelihood ratio to estimate the quark fraction in a synthetic mixed quark/gluon sample, showing that the resultant uncertainties empirically satisfy the desired coverage properties.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (114)

  1. A. M. Sirunyan et al. (CMS Collaboration), Identification of heavy-flavour jets with the CMS detector in pp collisions at 13 TeV, J. Instrum. 13, P05011 (2018).
  2. A. Butter et al., The machine learning landscape of top taggers, SciPost Phys. 7, 014 (2019).
  3. E. Bols, J. Kieseler, M. Verzetti, M. Stoye, and A. Stakia, Jet flavour classification using DeepJet, J. Instrum. 15, P12012 (2020).
  4. H. Qu, C. Li, and S. Qian, Particle transformer for jet tagging, arXiv:2202.03772.
  5. S. Gong, Q. Meng, J. Zhang, H. Qu, C. Li, S. Qian, W. Du, Z.-M. Ma, and T.-Y. Liu, An efficient Lorentz equivariant graph neural network for jet tagging, J. High Energy Phys. 07 (2022) 030.
  6. A. Bogatskiy, T. Hoffman, D. W. Miller, J. T. Offermann, and X. Liu, Explainable equivariant neural networks for particle physics: PELICAN, J. High Energy Phys. 03 (2024) 113.
  7. G. Aad et al. (ATLAS Collaboration), Dijet resonance search with weak supervision using s=13  TeV pp collisions in the ATLAS detector, Phys. Rev. Lett. 125, 131801 (2020).
  8. G. Kasieczka et al., The LHC Olympics 2020 a community challenge for anomaly detection in high energy physics, Rep. Prog. Phys. 84, 124201 (2021).
  9. T. Aarrestad et al., The dark machines anomaly score challenge: Benchmark data and model independent event classification for the Large Hadron Collider, SciPost Phys. 12, 043 (2022).
  10. E. M. Metodiev, J. Thaler, and R. Wynne, Anomaly detection in collider physics via factorized observables, Phys. Rev. D 110, 055012 (2024).
  11. G. Aad et al. (ATLAS Collaboration), Anomaly detection search for new resonances decaying into a Higgs boson and a generic new particle X in hadronic final states using s=13  TeV pp collisions with the ATLAS detector, Phys. Rev. D 108, 052009 (2023).
  12. G. Aad et al. (ATLAS Collaboration), Search for new phenomena in two-body invariant mass distributions using unsupervised machine learning for anomaly detection at s = 13 TeV with the ATLAS detector, Phys. Rev. Lett. 132, 081801 (2024).
  13. V. Chekhovsky et al. (CMS Collaboration), Model-agnostic search for dijet resonances with anomalous jet substructure in proton–proton collisions at s=13  TeV, Rep. Prog. Phys. 88, 067802 (2025).
  14. R. Gambhir, R. Mastandrea, B. Nachman, and J. Thaler, Isolating unisolated Upsilons with anomaly detection in CMS open data, Phys. Rev. Lett. 135, 021902 (2025).
  15. G. Aad et al. (ATLAS Collaboration), Search for non-resonant Higgs boson pair production in the bbℓνℓν final state with the ATLAS detector in pp collisions at s=13  TeV, Phys. Lett. B 801, 135145 (2020).
  16. G. Aad et al. (ATLAS Collaboration), Search for Higgs boson decays into a Z boson and a light hadronically decaying resonance using 13 TeV pp collision data from the ATLAS detector, Phys. Rev. Lett. 125, 221802 (2020).
  17. G. Aad et al. (ATLAS Collaboration), Search for heavy resonances decaying into a pair of Z bosons in the ℓ+ℓ−ℓ′+ℓ′− and ℓ+ℓ−νν¯ final states using 139  fb−1 of proton–proton collisions at s=13  TeV with the ATLAS detector, Eur. Phys. J. C 81, 332 (2021).
  18. A. Tumasyan et al. (CMS Collaboration), Search for a heavy Higgs boson decaying into two lighter Higgs bosons in the ττbb final state at 13 TeV, J. High Energy Phys. 11 (2021) 057.
  19. A. Andreassen, P. T. Komiske, E. M. Metodiev, B. Nachman, and J. Thaler, OmniFold: A method to simultaneously unfold all observables, Phys. Rev. Lett. 124, 182001 (2020).
  20. M. Bellagente, A. Butter, G. Kasieczka, T. Plehn, and R. Winterhalder, How to GAN away detector effects, SciPost Phys. 8, 070 (2020).
  21. M. Bellagente, A. Butter, G. Kasieczka, T. Plehn, A. Rousselot, R. Winterhalder, L. Ardizzone, and U. Köthe, Invertible networks or partons to detector and back again, SciPost Phys. 9, 074 (2020).
  22. M. Vandegar, M. Kagan, A. Wehenkel, and G. Louppe, Neural empirical bayes: Source distribution estimation and its applications to simulation-based inference, arXiv:2011.05836.
  23. A. Andreassen, P. T. Komiske, E. M. Metodiev, B. Nachman, A. Suresh, and J. Thaler, Scaffolding simulations with deep learning for high-dimensional deconvolution, in Proceedings of the 9th International Conference on Learning Representations (2021).
  24. M. Arratia et al., Publishing unbinned differential cross section results, J. Instrum. 17, P01024 (2022).
  25. J. N. Howard, S. Mandt, D. Whiteson, and Y. Yang, Learning to simulate high energy particle collisions from unlabeled data, Sci. Rep. 12, 7567 (2022).
  26. M. Backes, A. Butter, M. Dunford, and B. Malaescu, An unfolding method based on conditional invertible neural networks (cINN) using iterative training, SciPost Phys. Core 7, 007 (2024).
  27. A. Shmakov, K. Greif, M. Fenton, A. Ghosh, P. Baldi, and D. Whiteson, End-to-end latent variational diffusion models for inverse problems in high energy physics, arXiv:2305.10399.
  28. A. Shmakov, K. Greif, M. J. Fenton, A. Ghosh, P. Baldi, and D. Whiteson, Full event particle-level unfolding with variable-length latent variational diffusion, SciPost Phys. 18, 117 (2025).
  29. S. Otten, S. Caron, W. de Swart, M. van Beekveld, L. Hendriks, C. van Leeuwen, D. Podareanu, R. Ruiz de Austri, and R. Verheyen, Event generation and statistical sampling for physics with deep generative models and a density information buffer, Nat. Commun. 12, 2985 (2021).
  30. B. Hashemi, N. Amin, K. Datta, D. Olivito, and M. Pierini, LHC analysis-specific datasets with generative adversarial networks, arXiv:1901.05282.
  31. R. Di Sipio, M. Faucci Giannelli, S. Ketabchi Haghighat, and S. Palazzo, DijetGAN: A generative-adversarial network approach for the simulation of QCD Dijet events at the LHC, J. High Energy Phys. 08 (2019) 110.
  32. A. Butter, T. Plehn, and R. Winterhalder, How to GAN LHC events, SciPost Phys. 7, 075 (2019).
  33. Y. Alanazi et al., Simulation of electron-proton scattering events by a Feature-Augmented and Transformed Generative Adversarial Network (FAT-GAN), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence Main Track, 10.24963/ijcai.2021/293 (2020), pp. 2126–2132.
  34. A. Butter and T. Plehn, Generative networks for LHC events, arXiv:2008.08558.
  35. A. Butter, T. Heimel, S. Hummerich, T. Krebs, T. Plehn, A. Rousselot, and S. Vent, Generative networks for precision enthusiasts, SciPost Phys. 14, 078 (2023).
  36. E. Buhmann, G. Kasieczka, and J. Thaler, EPiC-GAN: Equivariant point cloud generation for particle jets, SciPost Phys. 15, 130 (2023).
  37. O. Amram et al., CaloChallenge 2022: A community challenge for fast calorimeter simulation, arXiv:2410.21611.
  38. M. Feickert and B. Nachman, A living review of machine learning for particle physics, arXiv:2102.02770.
  39. J. Neyman and E. S. Pearson, On the problem of the most efficient tests of statistical hypotheses, Phil. Trans. R. Soc. A 231, 289 (1933).
  40. K. Cranmer, J. Pavez, and G. Louppe, Approximating likelihood ratios with calibrated discriminative classifiers, arXiv:1506.02169.
  41. J. Hermans, A. Delaunoy, F. Rozet, A. Wehenkel, V. Begy, and G. Louppe, A trust crisis in simulation-based inference? Your posterior approximations can be unfaithful, arXiv:2110.06581.
  42. J. Neyman, Outline of a theory of statistical estimation based on the classical theory of probability, Phil. Trans. R. Soc. A 236, 333 (1937).
  43. P. J. Huber, Robust estimation of a location parameter, Ann. Math. Stat. 35, 73 (1964).
  44. G. Gong and F. J. Samaniego, Pseudo maximum likelihood estimation: Theory and applications, Ann. Stat. 9, 861 (1981).
  45. S. Navas et al. (Particle Data Group), Review of particle physics, Phys. Rev. D 110, 030001 (2024).
  46. G. P. Salam, The strong coupling: A theoretical perspective, in From My Vast Repertoire …: Guido Altarelli’s Legacy, edited by A. Levy, S. Forte, and G. Ridolfi (World Scientific, Singapore, 2019), pp. 101–121.
  47. A. Pich, J. Rojo, R. Sommer, and A. Vairo, Determining the strong coupling: status and challenges, Proc. Sci., Confinement2018 (2018) 035 [arXiv:1811.11801].
  48. D. d’Enterria et al., αs(2019): Precision measurements of the QCD Coupling, arXiv:1907.01435.
  49. A. Pich, Precision physics with inclusive QCD processes, Prog. Part. Nucl. Phys. 117, 103846 (2021).
  50. D. d’Enterria et al., The strong coupling constant: State of the art and the decade ahead, J. Phys. G 51, 090501 (2024).
  51. P. Nason, The top mass in hadronic collisions, in From My Vast Repertoire …: Guido Altarelli’s Legacy, edited by A. Levy, S. Forte, and G. Ridolfi (World Scientific, Singapore, 2019) pp. 123–151.
  52. A. H. Hoang, What is the top quark mass?, Annu. Rev. Nucl. Part. Sci. 70, 225 (2020).
  53. A. Hayrapetyan et al. (ATLAS Collaboration and CMS Collaboration), Combination of measurements of the top quark mass from data collected by the ATLAS and CMS experiments at s=7 and 8 TeV, Phys. Rev. Lett. 132, 261902 (2024).
  54. B. Lakshminarayanan, A. Pritzel, and C. Blundell, Simple and scalable predictive uncertainty estimation using deep ensembles, arXiv:1612.01474.
  55. T. G. Dietterich, Ensemble methods in machine learning, in Multiple Classifier Systems (Springer, Berlin, Heidelberg, 2000), pp. 1–15.
  56. G. Aad et al. (ATLAS Collaboration), An implementation of neural simulation-based inference for parameter estimation in ATLAS, Rep. Prog. Phys. 88, 067801 (2025).
  57. J. Arbel, K. Pitas, M. Vladimirova, and V. Fortuin, A primer on bayesian neural networks: Review and debates, arXiv:2309.16314.
  58. C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra, Weight uncertainty in neural networks, arXiv:1505.05424.
  59. A. Graves, Practical variational inference for neural networks, in Advances in Neural Information Processing Systems, Vol. 24, edited by J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger (Curran Associates, Inc., Red Hook, NY, USA, 2011).
  60. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, Dropout: A simple way to prevent neural networks from overfitting, J. Mach. Learn. Res. 15, 1929 (2014).
  61. Y. Gal and Z. Ghahramani, Dropout as a bayesian approximation: Representing model uncertainty in deep learning, in Proceedings of The 33rd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 48, edited by M. F. Balcan and K. Q. Weinberger (PMLR, New York, 2016), pp. 1050–1059.
  62. F. D’Angelo and V. Fortuin, Repulsive deep ensembles are bayesian, arXiv:2106.11642.
  63. S. S. Wilks, The large-sample distribution of the likelihood ratio for testing composite hypotheses, Ann. Math. Stat. 9, 60 (1938).
  64. N. Dalmasso, R. Izbicki, and A. B. Lee, Confidence sets and hypothesis testing in a likelihood-free inference setting, arXiv:2002.10399.
  65. N. Dalmasso, L. Masserano, D. Zhao, R. Izbicki, and A. B. Lee, Likelihood-free frequentist inference: Bridging classical statistics and machine learning for reliable simulator-based inference, Electron. J. Stat. 18, 5045 (2024).
  66. M. Sugiyama, T. Suzuki, S. Nakajima, H. Kashima, P. von Bünau, and M. Kawanabe, Direct importance estimation for covariate shift adaptation, Ann. Inst. Stat. Math. 60, 699 (2008).
  67. Y. Freund and R. E. Schapire, A decision-theoretic generalization of on-line learning and an application to boosting, J. Comput. Syst. Sci. 55, 119 (1997).
  68. G. Lebanon and J. Lafferty, Boosting and maximum likelihood for exponential models, in Advances in Neural Information Processing Systems, Vol. 14, edited by T. Dietterich, S. Becker, and Z. Ghahramani (MIT Press, Cambridge, MA, 2001).
  69. G. Louppe, Understanding random forests: From theory to practice, arXiv:1407.7502.
  70. F. Wenzel, J. Snoek, D. Tran, and R. Jenatton, Hyperparameter ensembles for robustness and uncertainty quantification, arXiv:2006.13570.
  71. B. Nachman and J. Thaler, Learning from many collider events at once, Phys. Rev. D 103, 116013 (2021).
  72. R. T. D’Agnolo and A. Wulzer, Learning new physics from a machine, Phys. Rev. D 99, 015014 (2019).
  73. R. T. D’Agnolo, G. Grosso, M. Pierini, A. Wulzer, and M. Zanetti, Learning multivariate new physics, Eur. Phys. J. C 81, 89 (2021).
  74. R. Gambhir, B. Nachman, and J. Thaler, Bias and priors in machine learning calibrations for high energy physics, Phys. Rev. D 106, 036011 (2022).
  75. R. Gambhir, B. Nachman, and J. Thaler, Learning uncertainties the frequentist way: Calibration and correlation in high energy physics, Phys. Rev. Lett. 129, 082001 (2022).
  76. D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, arXiv:1412.6980.
  77. A. Paszke et al., pytorch: An imperative style, high-performance deep learning library, arXiv:1912.01703.
  78. R. Feinman, pytorch-minimize (2022).
  79. P. Virtanen et al. (SciPy 1.0 Contributors), SciPy 1.0: Fundamental algorithms for scientific computing in Python, Nat. Methods 17, 261 (2020).
  80. L. M. Jones, Tests for determining the parton ancestor of a hadron jet, Phys. Rev. D 39, 2550 (1989).
  81. Z. Fodor, How to see the differences between quark and gluon jets, Phys. Rev. D 41, 1726 (1990).
  82. L. Lonnblad, C. Peterson, and T. Rognvaldsson, Finding gluon jets with a neural trigger, Phys. Rev. Lett. 65, 1321 (1990).
  83. L. Lonnblad, C. Peterson, and T. Rognvaldsson, Using neural networks to identify jets, Nucl. Phys. B349, 675 (1991).
  84. I. Csabai, F. Czako, and Z. Fodor, Quark and gluon jet separation using neural networks, Phys. Rev. D 44, R1905 (1991).
  85. L. Jones, Towards a systematic jet classification, Phys. Rev. D 42, 811 (1990).
  86. J. Pumplin, How to tell quark jets from gluon jets, Phys. Rev. D 44, 2025 (1991).
  87. P. D. Acton et al. (OPAL Collaboration), A study of differences between quark and gluon jets using vertex tagging of quark jets, Z. Phys. C 58, 387 (1993).
  88. J. Gallicchio and M. D. Schwartz, Quark and gluon jet substructure, J. High Energy Phys. 04 (2013) 090.
  89. P. Gras, S. Höche, D. Kar, A. Larkoski, L. Lönnblad, S. Plätzer, A. Siódmok, P. Skands, G. Soyez, and J. Thaler, Systematics of quark/gluon tagging, J. High Energy Phys. 07 (2017) 091.
  90. A. J. Larkoski, I. Moult, and B. Nachman, Jet substructure at the Large Hadron Collider: A review of recent advances in theory and machine learning, Phys. Rep. 841, 1 (2020).
  91. R. Kogler et al., Jet substructure at the Large Hadron Collider: Experimental review, Rev. Mod. Phys. 91, 045003 (2019).
  92. P. Komiske, E. Metodiev, and J. Thaler, Pythia8 quark and gluon jets for energy flow, Zenodo, 10.5281/zenodo.3164691 (2019).
  93. P. T. Komiske, E. M. Metodiev, and J. Thaler, Energy flow networks: Deep sets for particle jets, J. High Energy Phys. 01 (2019) 121.
  94. C. Bierlich et al., A comprehensive guide to the physics and usage of PYTHIA 8.3, SciPost Phys. Codebases 2022, 8 (2022).
  95. M. Cacciari, G. P. Salam, and G. Soyez, The anti-kt jet clustering algorithm, J. High Energy Phys. 04 (2008) 063.
  96. M. Cacciari, G. P. Salam, and G. Soyez, fastjet user manual, Eur. Phys. J. C 72, 1896 (2012).
  97. S. Bright-Thonney, B. Nachman, and J. Thaler, Infrared-safe energy weighting does not guarantee small nonperturbative effects, Phys. Rev. D 110, 014029 (2024).
  98. C. F. Berger, T. Kucs, and G. F. Sterman, Event shape/ energy flow correlations, Phys. Rev. D 68, 014012 (2003).
  99. L. G. Almeida, S. J. Lee, G. Perez, G. F. Sterman, I. Sung, and J. Virzi, Substructure of high-pT jets at the LHC, Phys. Rev. D 79, 074017 (2009).
  100. S. D. Ellis, C. K. Vermilion, J. R. Walsh, A. Hornig, and C. Lee, Jet shapes and jet algorithms in SCET, J. High Energy Phys. 11 (2010) 101.
  101. A. J. Larkoski, J. Thaler, and W. J. Waalewijn, Gaining (mutual) information about quark/gluon discrimination, J. High Energy Phys. 11 (2014) 129.
  102. G. Huang, Y. Li, G. Pleiss, Z. Liu, J. E. Hopcroft, and K. Q. Weinberger, Snapshot ensembles: Train 1, get m for free, arXiv:1704.00109.
  103. T. Garipov, P. Izmailov, D. Podoprikhin, D. Vetrov, and A. G. Wilson, Loss surfaces, mode connectivity, and fast ensembling of DNNs, arXiv:1802.10026.
  104. Y. Wen, D. Tran, and J. Ba, Batchensemble: An alternative approach to efficient ensemble and lifelong learning, arXiv:2002.06715.
  105. F. T. Acosta, T. Wamorkar, V. Mikuni, and B. Nachman, Stabilizing neural likelihood ratio estimation, arXiv:2503.20753.
  106. S. Benevedes and J. Thaler, wifi ensemble code, Github (2025).
  107. E. Slutsuky, Über stochastische asymptoten und grenzwerte (abdruck), Tohoku Math. J. 27, 67 (1925).
  108. A. Wald, Tests of statistical hypotheses concerning several parameters when the number of observations is large, Trans. Am. Math. Soc. 54, 426 (1943).
  109. Y. Chen and K.-Y. Liang, On the asymptotic behaviour of the pseudolikelihood ratio test statistic with boundary problems, Biometrika 97, 603 (2010).
  110. D. R. Cox and E. J. Snell, A general definition of residuals, J. R. Stat. Soc. Ser. B 30, 248 (1968).
  111. E. Naftali, First order bias and second order variance of the maximum likelihood estimator with application to multivariate Gaussian data and time delay and Doppler shift estimation, Ph.D. thesis, MIT, 2000.
  112. P. Sur and E. J. Candès, A modern maximum-likelihood theory for high-dimensional logistic regression, Proc. Natl. Acad. Sci. U.S.A. 116, 14516 (2019).
  113. P. Rilstone, V. Srivastava, and A. Ullah, The second-order bias and mean squared error of nonlinear estimators, J. Econ. 75, 369 (1996).
  114. K. I. Kim, Higher order bias correcting moment equation for m-estimation and its higher order efficiency, Econometrics 4, 48 (2016).

Outline

Information

Phys. Rev. D 112, 056024– Published 19 September, 2025
  • Received 7 June 2025
  • Accepted 3 September 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI. Funded by SCOAP3.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation