- Physical Review Journals
All JournalsPhysics Magazine

Physical Review D

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Calculation of p-values for quadratic statistics in pulsar timing arrays

Rutger van Haasteren*

  • *Contact author: rutger@vhaasteren.com
PDF

Phys. Rev. D 112, 103009 – Published 6 November, 2025

DOI: https://doi.org/10.1103/b3y4-yfr1

Abstract

Pulsar Timing Array (PTA) projects have reported various lines of evidence suggesting the presence of a stochastic gravitational wave (GW) background in their data. One key line of evidence involves a detection statistic sensitive to interpulsar correlations, such as those induced by GWs. A p-value is then calculated to assess how unlikely it is for the observed signal to arise under the null hypothesis H0, purely by chance. However, PTAs cannot empirically draw samples from H0. As a workaround, various techniques are used in the literature to approximate p-values under H0. One such technique, which has been characterized as a model-independent method, is the use of “scrambling” transformations that modify the data to cancel out pulsar correlations, thereby simulating realizations from H0. In this work, scrambling methods and the detection statistic are investigated from first principles. The p-value methodology that is discussed is general, but the discussions regarding a specific detection statistic apply to the detection of a stochastic background of gravitational waves with PTAs. All methods in the literature to calculate p-values for such a detection statistic are rigorously analyzed, and analytical expressions are derived for the distribution of the detection statistic and the corresponding p-values. All this leads to the conclusion that scrambling methods are not model independent and thus not completely empirical. With a single realization of data our results are necessarily always model dependent, which any analysis will need to accept. Instead of scrambling approaches, rigorous Bayesian and Frequentist p-value calculation methods are advocated, the evaluation of which depend on the generalized χ2 distribution. This view is consistent with the posterior predictive p-value approach that is already in the literature. Efficient expressions are derived to evaluate the generalized χ2 distribution of the detection statistic on real data. It is highlighted that no Frequentist p-values have been calculated correctly in the PTA literature to date.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (61)

  1. G. Agazie et al. (The NANOGrav Collaboration), The NANOGrav 15 Yr data set: Evidence for a gravitational-wave background, Astrophys. J. Lett. 951, L8 (2023).
  2. D. J. Reardon et al., Search for an isotropic gravitational-wave background with the Parkes Pulsar Timing Array, Astrophys. J. Lett. 951, L6 (2023).
  3. J. Antoniadis et al., The second data release from the European Pulsar Timing Array II. Customised pulsar noise models for spatially correlated gravitational waves, Astron. Astrophys. 678, A49 (2023).
  4. H. Xu et al., Searching for the Nano-Hertz stochastic gravitational wave background with the Chinese Pulsar Timing Array data release I, Res. Astron. Astrophys. 23, 075024 (2023).
  5. G. Agazie et al. (The NANOGrav Collaboration), The NANOGrav 15 Yr data set: Bayesian limits on gravitational waves from individual supermassive black hole binaries, Astrophys. J. Lett. 951, L50 (2023).
  6. J. Antoniadis et al., The second data release from the European Pulsar Timing Array—IV. Implications for massive black holes, dark matter, and the Early Universe, Astron. Astrophys. 685, A94 (2024).
  7. J. Antoniadis et al., The second data release from the European Pulsar Timing Array V. Search for continuous gravitational wave signals, Astron. Astrophys. 690, A118 (2024).
  8. A. Afzal et al., The NANOGrav Collaboration, The NANOGrav 15 Yr data set: Search for signals from new physics, Astrophys. J. Lett. 951, L11 (2023).
  9. B. Allen, S. Dhurandhar, Y. Gupta, M. McLaughlin, P. Natarajan, R. M. Shannon, E. Thrane, and A. Vecchio, The International Pulsar Timing Array checklist for the detection of nanohertz gravitational waves, arXiv:2304.04767.
  10. J. S. Hazboun, P. M. Meyers, J. D. Romano, X. Siemens, and A. M. Archibald, Analytic distribution of the optimal cross-correlation statistic for stochastic gravitational-wave-background searches using Pulsar Timing Arrays, Phys. Rev. D 108, 104050 (2023).
  11. M. Vallisneri, P. M. Meyers, K. Chatziioannou, and A. J. K. Chua, Posterior predictive checking for gravitational-wave detection with Pulsar Timing Arrays. I. The optimal statistic, Phys. Rev. D 108, 123007 (2023).
  12. R. van Haasteren, Pulsar Timing Arrays require hierarchical models, Astrophys. J. Suppl. Ser. 273, 23 (2024).
  13. G. Agazie et al. (The NANOGrav Collaboration), The NANOGrav 15 Yr data set: Observations and timing of 68 millisecond pulsars, Astrophys. J. Lett. 951, L9 (2023).
  14. C. Pernet, Null hypothesis significance testing: A short tutorial, F1000Res. 4, 621 (2015).
  15. R. Nuzzo, Scientific method: Statistical errors, Nature (London) 506, 150 (2014).
  16. R. L. Wasserstein and N. A. Lazar, The ASA statement on P-values: Contextprocess, and purpose, Am. Stat. 70, 129 (2016).
  17. D. J. Benjamin et al., Redefine statistical significance, Nat. Hum. Behav. 2, 6 (2018).
  18. R. L. Wasserstein, Allen L. Schirm, and N. A. Lazar, Moving to a world beyond “p<0.05”, Am. Stat. 73, 1 (2019).
  19. F. A. Jenet, G. B. Hobbs, K. J. Lee, and R. N. Manchester, Detecting the stochastic gravitational wave background using pulsar timing, Astrophys. J. 625, L123 (2005).
  20. M. Anholm, S. Ballmer, J. D. E. Creighton, L. R. Price, and X. Siemens, Optimal strategies for gravitational wave stochastic background searches in pulsar timing data, Phys. Rev. D 79, 084030 (2009).
  21. S. J. Chamberlin, J. D. E. Creighton, X. Siemens, P. Demorest, J. Ellis, L. R. Price, and J. D. Romano, Time-domain implementation of the optimal cross-correlation statistic for stochastic gravitational-wave background searches in pulsar timing data, Phys. Rev. D 91, 044048 (2015).
  22. B. Allen and J. D. Romano, Hellings and downs correlation of an arbitrary set of pulsars, Phys. Rev. D 108, 043026 (2023).
  23. K. A. Gersbach, S. R. Taylor, P. M. Meyers, and J. D. Romano, Spatial and spectral characterization of the gravitational-wave background with the PTA optimal statistic, Phys. Rev. D 111, 023027 (2025).
  24. M. Tegmark, How to measure CMB power spectra without losing information, Phys. Rev. D 55, 5895 (1997).
  25. M. Tegmark, A. N. Taylor, and A. F. Heavens, Karhunen-Loève eigenvalue problems in cosmology: How should we tackle large data sets?, Astrophys. J. 480, 22 (1997).
  26. S. C. Sardesai, S. J. Vigeland, K. A. Gersbach, and S. R. Taylor, Generalized optimal statistic for characterizing multiple correlated signals in pulsar timing arrays, Phys. Rev. D 108, 124081 (2023).
  27. S. J. Vigeland, K. Islo, S. R. Taylor, and J. A. Ellis, Noise-marginalized optimal statistic: A robust hybrid Frequentist-Bayesian statistic for the stochastic gravitational-wave background in pulsar timing arrays, Phys. Rev. D 98, 044003 (2018).
  28. J. Neyman, E. S. Pearson, and K. Pearson, IX. On the problem of the most efficient tests of statistical hypotheses, Phil. Trans. R. Soc. A 231, 289 (1997).
  29. P. A. Rosado, A. Sesana, and J. Gair, Expected properties of the first gravitational wave signal detected with pulsar timing arrays, Mon. Not. R. Astron. Soc. 451, 2417 (2015).
  30. R. van Haasteren, B. Allen, and J. D. Romano, Optimal robust detection statistics for pulsar timing arrays, arXiv:2509.06489.
  31. J. P. Imhof, Computing the distribution of quadratic forms in normal variables, Biometrika 48, 419 (1961).
  32. A. Das and W. S. Geisler, A method to integrate and classify normal distributions, J. Vis. 21, 1 (2021).
  33. N. J. Cornish and L. Sampson, Towards robust gravitational wave detection with pulsar timing arrays, Phys. Rev. D 93, 104047 (2016).
  34. S. R. Taylor, L. Lentati, S. Babak, P. Brem, J. R. Gair, A. Sesana, and A. Vecchio, All correlations must die: Assessing the significance of a stochastic gravitational-wave background in pulsar timing arrays, Phys. Rev. D 95, 042002 (2017).
  35. V. Di Marco, A. Zic, M. T. Miles, D. J. Reardon, E. Thrane, and R. M. Shannon, Toward robust detections of nanohertz gravitational waves, Astrophys. J. 956, 14 (2023).
  36. M. T. Miles et al., The MeerKAT pulsar timing array: The first search for gravitational waves with the MeerKAT radio telescope, Mon. Not. R. Astron. Soc. 536, 1489 (2024).
  37. F. Mezzadri, How to generate random matrices from the classical compact groups, arXiv:math-ph/0609050.
  38. J. Antoniadis, S. Babak, A.-S. B. Nielsen, C. G. Bassa, and A. Berthereau, The second data release from the European Pulsar Timing Array. I. The dataset and timing analysis, Astron. Astrophys. 678, A48 (2023).
  39. A. Zic et al., The Parkes Pulsar Timing Array third data release, Pub. Astron. Soc. Aust. 40, e049 (2023).
  40. L. Isserlis, On a formula for the product-moment coefficient of any order of a normal frequency distribution in any number of variables, Biometrika 12, 134 (1918).
  41. J. Antoniadis et al., The second data release from the European Pulsar Timing Array—III. Search for gravitational wave signals, Astron. Astrophys. 678, A50 (2023).
  42. R. van Haasteren, Use model averaging instead of model selection in pulsar timing, Mon. Not. R. Astron. Soc.: Lett. 537, L1 (2025).
  43. B. P. Abbott et al. (LIGO Scientific Collaboration and Virgo Collaboration), Observation of gravitational waves from a binary black hole merger, Phys. Rev. Lett. 116, 061102 (2016).
  44. M. T. Miles et al., The MeerKAT pulsar timing array: The 4.5-year data release and the noise and stochastic signals of the millisecond pulsar population, Mon. Not. R. Astron. Soc. 536, 1467 (2024).
  45. G. E. P. Box, Sampling and Bayes’ inference in scientific modelling and robustness, J. R. Stat. Soc. A 143, 383 (1980).
  46. G. Agazie et al., The NANOGrav 15 Yr data set: Posterior predictive checks for gravitational-wave detection with pulsar timing arrays, Phys. Rev. D 111, 042011 (2025).
  47. R. van Haasteren and M. Vallisneri, New advances in the Gaussian-process approach to pulsar-timing data analysis, Phys. Rev. D 90, 104012 (2014).
  48. R. van Haasteren and M. Vallisneri, Low-rank approximations for large stationary covariance matrices, as used in the Bayesian and generalized-least-squares analysis of pulsar-timing data, Mon. Not. R. Astron. Soc. 446, 1170 (2015).
  49. J. Sherman and W. J. Morrison, Adjustment of an inverse matrix corresponding to a change in one element of a given matrix, Ann. Math. Stat. 21, 124 (1950).
  50. M. Woodbury, Inverting Modified Matrices, Memorandum Report/Statistical Research Group, Princeton (Department of Statistics, Princeton University, 1950).
  51. J. A. Ellis, M. Vallisneri, S. R. Taylor, and P. T. Baker, enterprise: Enhanced Numerical Toolbox Enabling a Robust PulsaR Inference SuitE, Zenodo, 10.5281/zenodo.4059815 (2020).
  52. Z. Arzoumanian et al., The NANOGrav nine-year data set: Observations, Aarrival time measurements, and analysis of 37 millisecond pulsars, Astrophys. J. 813, 65 (2015).
  53. M. Seeger, Low Rank Updates for the Cholesky Decomposition, Tech. Rep. (University of California at Berkeley, Berkeley, 2004).
  54. R. van Haasteren, fastshermanmorrison: Fast Sherman-Morrison updates for pulsar timing analysis, Zenodo, 10.5281/zenodo.17439723 (2025).
  55. Z. Arzoumanian et al., The NANOGrav nine-year data set: Limits on the isotropic stochastic gravitational wave background, Astrophys. J. 821, 13 (2016).
  56. A. D. Johnson et al., The NANOGrav 15-year gravitational-wave background analysis pipeline, Phys. Rev. D 109, 103012 (2024).
  57. https://github.com/vhaasteren/p-values-figures.
  58. A. Haar, Der massbegriff in der theorie der kontinuierlichen gruppen, Ann. Math. 34, 147 (1933).
  59. B. Collins and P. Śniady, Integration with respect to the haar measure on unitary, orthogonal and symplectic group, Commun. Math. Phys. 264, 773 (2006).
  60. P. Virtanen et al., scipy 1.0: Fundamental algorithms for scientific computing in python, Nat. Methods 17, 261 (2020).
  61. C. R. Harris et al., Array programming with numpy, Nature (London) 585, 357 (2020).

Outline

Information

Phys. Rev. D 112, 103009– Published 6 November, 2025
  • Received 13 June 2025
  • Accepted 22 September 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI. Open access publication funded by the Max Planck Society.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation