- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Letter
  • Open Access

Convolutional transformer wave functions

Ao Chen1,2,*,†, Vighnesh Dattatraya Naik1,*,‡, and Markus Heyl1

  • *These authors contributed equally to this work.
  • †Contact author: chenao.phys@gmail.com
  • ‡Contact author: naikvighnesh123@gmail.com
PDF

Phys. Rev. Research 8, L022040 – Published 3 June, 2026

DOI: https://doi.org/10.1103/7xwp-25y9

Abstract

Deep neural quantum states have recently achieved remarkable performance in solving challenging quantum many-body problems. While transformer networks appear particularly promising due to their success in computer science, we show that previously reported transformer wave functions have not so far been capable of utilizing their full power. Here, we introduce the convolutional transformer wave function (CTWF). We show that our CTWF exhibits state-of-the-art performance in ground-state search and nonequilibrium dynamics compared to previous results, demonstrating its promising capacity in complex quantum problems.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (57)

  1. G. Carleo and M. Troyer, Solving the quantum many-body problem with artificial neural networks, Science 355, 602 (2017).
  2. Y. Nomura and M. Imada, Dirac-type nodal spin liquid revealed by refined quantum many-body solver using neural-network wave function, correlation ratio, and level spectroscopy, Phys. Rev. X 11, 031034 (2021).
  3. N. Astrakhantsev, T. Westerhout, A. Tiwari, K. Choo, A. Chen, M. H. Fischer, G. Carleo, and T. Neupert, Broken-symmetry ground states of the Heisenberg model on the pyrochlore lattice, Phys. Rev. X 11, 041021 (2021).
  4. A. Chen and M. Heyl, Empowering deep neural quantum states through efficient optimization, Nat. Phys. 20, 1476 (2024).
  5. Y. Nomura, A. S. Darmawan, Y. Yamaji, and M. Imada, Restricted Boltzmann machine learning for solving strongly correlated quantum systems, Phys. Rev. B 96, 205152 (2017).
  6. D. Luo and B. K. Clark, Backflow transformations via neural networks for quantum many-body wave functions, Phys. Rev. Lett. 122, 226401 (2019).
  7. J. R. Moreno, G. Carleo, A. Georges, and J. Stokes, Fermionic wave functions from neural-network constrained hidden states, Proc. Natl. Acad. Sci. USA 119, e2122059119 (2022).
  8. J. Hermann, J. Spencer, K. Choo, A. Mezzacapo, W. M. C. Foulkes, D. Pfau, G. Carleo, and F. Noé, Ab initio quantum chemistry with neural-network wavefunctions, Nat. Rev. Chem. 7, 692 (2023).
  9. K. Choo, A. Mezzacapo, and G. Carleo, Fermionic neural-network states for ab-initio electronic structure, Nat. Commun. 11, 2368 (2020).
  10. D. Pfau, J. S. Spencer, A. G. D. G. Matthews, and W. M. C. Foulkes, Ab initio solution of the many-electron Schrödinger equation with deep neural networks, Phys. Rev. Res. 2, 033429 (2020).
  11. J. Hermann, Z. Schätzle, and F. Noé, Deep-neural-network solution of the electronic Schrödinger equation, Nat. Chem. 12, 891 (2020).
  12. D. Pfau, S. Axelrod, H. Sutterud, I. von Glehn, and J. S. Spencer, Accurate computation of quantum excited states with neural networks, Science 385, eadn0137 (2024).
  13. A. Nagy and V. Savona, Variational quantum Monte Carlo method with a neural-network ansatz for open quantum systems, Phys. Rev. Lett. 122, 250501 (2019).
  14. M. J. Hartmann and G. Carleo, Neural-network approach to dissipative quantum many-body dynamics, Phys. Rev. Lett. 122, 250502 (2019).
  15. F. Vicentini, A. Biella, N. Regnault, and C. Ciuti, Variational neural-network ansatz for steady states in open quantum systems, Phys. Rev. Lett. 122, 250503 (2019).
  16. M. Schmitt and M. Heyl, Quantum many-body dynamics in two dimensions with artificial neural networks, Phys. Rev. Lett. 125, 100503 (2020).
  17. M. Schmitt, M. M. Rams, J. Dziarmaga, M. Heyl, and W. H. Zurek, Quantum phase transition dynamics in the two-dimensional transverse-field Ising model, Sci. Adv. 8, eabl6850 (2022).
  18. A. Sinibaldi, C. Giuliani, G. Carleo, and F. Vicentini, Unbiasing time-dependent variational Monte Carlo by projected quantum evolution, Quantum 7, 1131 (2023).
  19. T. Mendes-Santos, M. Schmitt, and M. Heyl, Highly resolved spectral functions of two-dimensional systems with neural quantum states, Phys. Rev. Lett. 131, 046501 (2023).
  20. J. Nys, G. Pescia, A. Sinibaldi, and G. Carleo, Ab-initio variational wave functions for the time-dependent many-electron Schrödinger equation, Nat. Commun. 15, 9404 (2024).
  21. T. Mendes-Santos, M. Schmitt, A. Angelone, A. Rodriguez, P. Scholl, H. J. Williams, D. Barredo, T. Lahaye, A. Browaeys, M. Heyl, and M. Dalmonte, Wave-function network description and Kolmogorov complexity of quantum many-body systems, Phys. Rev. X 14, 021029 (2024).
  22. Y. Nomura, Helping restricted Boltzmann machines with quantum-state representation by restoring symmetry, J. Phys.: Condens. Matter 33, 174003 (2021).
  23. K. Choo, T. Neupert, and G. Carleo, Two-dimensional frustrated J1–J2 model studied with neural network quantum states, Phys. Rev. B 100, 125124 (2019).
  24. O. Sharir, Y. Levine, N. Wies, G. Carleo, and A. Shashua, Deep autoregressive models for the efficient variational simulation of many-body quantum systems, Phys. Rev. Lett. 124, 020503 (2020).
  25. M. Hibat-Allah, M. Ganahl, L. E. Hayward, R. G. Melko, and J. Carrasquilla, Recurrent neural network wave functions, Phys. Rev. Res. 2, 023358 (2020).
  26. C. Roth, A. Szabó, and A. H. MacDonald, High-accuracy variational Monte Carlo for frustrated magnets with deep neural networks, Phys. Rev. B 108, 054410 (2023).
  27. X. Liang, M. Li, Q. Xiao, J. Chen, C. Yang, H. An, and L. He, Deep learning representations for quantum many-body systems on heterogeneous hardware, Mach. Learn.: Sci. Technol. 4, 015035 (2023).
  28. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, Attention is all you need, in Advances in Neural Information Processing Systems, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Curran Associates, Inc., Red Hook, NY, 2017), Vol. 30.
  29. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, An image is worth 16×16 words: Transformers for image recognition at scale, in International Conference on Learning Representations (2021).
  30. I. von Glehn, J. S. Spencer, and D. Pfau, A self-attention ansatz for Ab-initio quantum chemistry, in International Conference on Learning Representations (2023). [OpenReview / ICLR Proceedings, Virtual Conference.].
  31. G. Pescia, J. Nys, J. Kim, A. Lovato, and G. Carleo, Message-passing neural quantum states for the homogeneous electron gas, Phys. Rev. B 110, 035108 (2024).
  32. H. Shang, C. Guo, Y. Wu, Z. Li, and J. Yang, Solving Schrödinger equation with a language model, arXiv:2307.09343.
  33. L. L. Viteritti, R. Rende, and F. Becca, Transformer variational wave functions for frustrated quantum spin systems, Phys. Rev. Lett. 130, 236401 (2023).
  34. Y.-H. Zhang and M. Di Ventra, Transformer quantum state: A multipurpose model for quantum many-body problems, Phys. Rev. B 107, 075147 (2023).
  35. H. Wu, B. Xiao, N. Codella, M. Liu, X. Dai, L. Yuan, and L. Zhang, CvT: Introducing convolutions to vision transformers, in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (IEEE, Piscataway, NJ, 2021), pp. 22–31.
  36. J. Guo, K. Han, H. Wu, Y. Tang, X. Chen, Y. Wang, and C. Xu, Cmt: Convolutional neural networks meet vision transformers, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, Piscataway, NJ, 2022), pp. 12175–12185.
  37. X. Cao, Z. Zhong, and Y. Lu, Vision transformer neural quantum states for impurity models, Phys. Rev. B 112, 235155 (2025).
  38. R. Rende and L. Loris Viteritti, Are queries and keys always relevant? A case study on transformer wave functions, Mach. Learn.: Sci. Technol. 6, 010501 (2025).
  39. K. Wu, H. Peng, M. Chen, J. Fu, and H. Chao, Rethinking and improving relative position encoding for vision transformer, in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (IEEE, Piscataway, NJ, 2021), pp. 10033–10041.
  40. R. Rende, L. L. Viteritti, L. Bardone, F. Becca, and S. Goldt, A simple linear algebra identity to optimize large-scale neural network quantum states, Commun. Phys. 7, 260 (2024).
  41. L. L. Viteritti, R. Rende, A. Parola, S. Goldt, and F. Becca, Transformer wave function for the Shastry-Sutherland model: Emergence of a spin-liquid phase, Phys. Rev. B 111, 134411 (2025).
  42. S. Roca-Jerat, M. Gallego, F. Luis, J. Carrete, and D. Zueco, Transformer wave function for quantum long-range models, Phys. Rev. B 110, 205147 (2024).
  43. V. Herráiz-López, S. Roca-Jerat, M. Gallego, R. Ferrández, J. Carrete, D. Zueco, and J. Román-Roche, First- and second-order quantum phase transitions in the long-range unfrustrated antiferromagnetic Ising chain, Phys. Rev. B 111, 014425 (2025).
  44. K. Sprague and S. Czischek, Variational Monte Carlo with large patched transformers, Commun. Phys. 7, 90 (2024).
  45. H. Lange, G. Bornet, G. Emperauger, C. Chen, T. Lahaye, S. Kienle, A. Browaeys, and A. Bohrdt, Transformer neural networks and quantum simulators: A hybrid approach for simulating strongly correlated systems, Quantum 9, 1675 (2024).
  46. D. Hendrycks and K. Gimpel, Gaussian error linear units (GELUs), arXiv:1606.08415.
  47. S. Sorella, Green function Monte Carlo with stochastic reconfiguration, Phys. Rev. Lett. 80, 4558 (1998).
  48. L. L. Viteritti, R. Rende, and F. Becca (private communication).
  49. D. Wu, R. Rossi, F. Vicentini, N. Astrakhantsev, F. Becca, X. Cao, J. Carrasquilla, F. Ferrari, A. Georges, M. Hibat-Allah et al., Variational benchmarks for quantum many-body problems, Science 386, 296 (2024).
  50. A. Sinibaldi, D. Hendry, F. Vicentini, and G. Carleo, Time-dependent neural Galerkin method for quantum dynamics, Phys. Rev. Lett. 136, 120402 (2026).
  51. L. Gravina, V. Savona, and F. Vicentini, Neural projected quantum dynamics: A systematic study, Quantum 9, 1803 (2024).
  52. H. W. J. Blöte and Y. Deng, Cluster Monte Carlo simulation of the transverse Ising model, Phys. Rev. E 66, 066110 (2002).
  53. Y. Gu, W. Li, H. Lin, B. Zhan, R. Li, Y. Huang, D. He, Y. Wu, T. Xiang, M. Qin, L. Wang, and D. Lv, Solving the Hubbard model with neural quantum states, arXiv:2507.02644.
  54. P. Weinberg and M. Bukov, QuSpin: A Python package for dynamics and exact diagonalisation of quantum many body systems. Part I: Spin chains, SciPost Phys. 2, 003 (2017).
  55. A. Chen and C. Roth, Quantax: Flexible neural quantum states based on QuSpin, JAX, and Equinox, 2025, https://github.com/ChenAo-Phys/quantax.
  56. M. Schmitt and M. Reh, jVMC: Versatile and performant variational Monte Carlo leveraging automated differentiation and GPU acceleration, SciPost Phys. Codebases 2 (2022).
  57. A. Chen, V. Naik, and M. Heyl, Convolutional transformer wave functions [Data set], Zenodo, 2024, https://doi.org/10.5281/zenodo.14035975.

Outline

Information

Phys. Rev. Research 8, L022040– Published 3 June, 2026
  • Received 30 April 2025
  • Accepted 15 May 2026
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation