- Open Access
Entanglement and optimization within autoregressive neural quantum states
Phys. Rev. Research 8, 013147 – Published 10 February, 2026
DOI: https://doi.org/10.1103/t2cg-kr7y
Abstract
Neural quantum states (NQSs) are powerful variational ansätze capable of representing highly entangled quantum many-body wavefunctions. While the average entanglement properties of ensembles of restricted Boltzmann machines are well understood, the entanglement structure of autoregressive NQSs such as recurrent neural networks and transformers remains largely unexplored. We perform large-scale simulations of ensembles of random autoregressive wavefunctions for chains of up to 256 spins and uncover signatures of transitions in their average entanglement scaling, entanglement spectra, and correlation functions. We show that the standard softmax normalization of the wavefunction suppresses entanglement and fluctuations, and introduce a square modulus normalization function that restores them. Finally, we connect the insights gained from our entanglement and activation function analysis to initialization strategies for finding the ground states of strongly correlated Hamiltonians via variational Monte Carlo.
Physics Subject Headings (PhySH)
Article Text
References (116)
- E. Y. Loh, Jr., J. E. Gubernatis, R. T. Scalettar, S. R. White, D. J. Scalapino, and R. L. Sugar, Sign problem in the numerical simulation of many-electron systems, Phys. Rev. B 41, 9301 (1990).
- A. W. Sandvik and J. Kurkijärvi, Quantum Monte Carlo simulation method for spin systems, Phys. Rev. B 43, 5950 (1991).
- P. Henelius and A. W. Sandvik, Sign problem in Monte Carlo simulations of frustrated quantum spin systems, Phys. Rev. B 62, 1102 (2000).
- M. Troyer and U.-J. Wiese, Computational complexity and fundamental limitations to fermionic quantum Monte Carlo simulations, Phys. Rev. Lett. 94, 170201 (2005).
- C. Wu and S.-C. Zhang, Sufficient condition for absence of the sign problem in the fermionic quantum Monte Carlo algorithm, Phys. Rev. B 71, 155115 (2005).
- Z.-X. Li, Y.-F. Jiang, and H. Yao, Solving the fermion sign problem in quantum Monte Carlo simulations by Majorana representation, Phys. Rev. B 91, 241117(R) (2015).
- Z. C. Wei, C. Wu, Y. Li, S. Zhang, and T. Xiang, Majorana positivity and the fermion sign problem of quantum Monte Carlo simulations, Phys. Rev. Lett. 116, 250601 (2016).
- R. Mondaini, S. Tarat, and R. T. Scalettar, Quantum critical points and the sign problem, Science 375, 418-424 (2022).
- P. Broecker and S. Trebst, Entanglement and the fermion sign problem in auxiliary field quantum Monte Carlo simulations, Phys. Rev. B 94, 075144 (2016).
- C. H. Mak and D. Chandler, Solving the sign problem in quantum Monte Carlo dynamics, Phys. Rev. A 41, 5709 (1990).
- U. Schollwöck, The density-matrix renormalization group in the age of matrix product states, Ann. Phys. 326, 96 (2011).
- F. Verstraete and J. I. Cirac, Matrix product states represent ground states faithfully, Phys. Rev. B 73, 094423 (2006).
- F. Verstraete, V. Murg, and J.I. Cirac, Matrix product states, projected entangled pair states, and variational renormalization group methods for quantum spin systems, Adv. Phys. 57, 143 (2008).
- L. Tagliacozzo, T. R. de Oliveira, S. Iblisdir, and J. I. Latorre, Scaling of entanglement support for matrix product states, Phys. Rev. B 78, 024410 (2008).
- X. Yu, D. Pekker, and B. K. Clark, Finding matrix product state representations of highly excited eigenstates of many-body localized Hamiltonians, Phys. Rev. Lett. 118, 017201 (2017).
- J. Tindall, A. Mello, M. Fishman, E. M. Stoudenmire, and D. Sels, Dynamics of disordered quantum systems with two- and three-dimensional tensor networks, arXiv:2503.05693.
- G. Carleo and M. Troyer, Solving the quantum many-body problem with artificial neural networks, Science 355, 602 (2017).
- G. Carleo, I. Cirac, K. Cranmer, L. Daudet, M. Schuld, N. Tishby, L. Vogt-Maranto, and L. Zdeborova, Machine learning and the physical sciences, Rev. Mod. Phys. 91, 045002 (2019).
- J. Carrasquilla, Machine learning for quantum matter, Adv. Phys.: X 5, 1797528 (2020).
- H. Lange, A. Van de Walle, A. Abedinnia, and A. Bohrdt, From architectures to applications: A review of neural quantum states, Quantum Sci. Technol. 9, 040501 (2024).
- A. Chen and M. Heyl, Empowering deep neural quantum states through efficient optimization, Nat. Phys. 20, 1476 (2024).
- M. Hibat-Allah, M. Ganahl, L. E. Hayward, R. G. Melko, and J. Carrasquilla, Recurrent neural network wave functions, Phys. Rev. Res. 2, 023358 (2020).
- M. Hibat-Allah, R. G. Melko, and J. Carrasquilla, Investigating topological order using recurrent neural networks, Phys. Rev. B 108, 075152 (2023).
- Z. Denis, A. Sinibaldi, and G. Carleo, Comment on “can neural quantum states learn volume-law ground states?” Phys. Rev. Lett. 134, 079701 (2025).
- J. Carrasquilla and R. G. Melko, Machine learning phases of matter, Nat. Phys. 13, 431 (2017).
- F. Schindler, N. Regnault, and T. Neupert, Probing many-body localization with neural networks, Phys. Rev. B 95, 245134 (2017).
- G. Torlai, G. Mazzola, J. Carrasquilla, M. Troyer, R. G. Melko, and G. Carleo, Neural-network quantum state tomography, Nat. Phys. 14, 447 (2018).
- M. Schmitt and M. Heyl, Quantum many-body dynamics in two dimensions with artificial neural networks, Phys. Rev. Lett. 125, 100503 (2020).
- J. Nys, Z. Denis, and G. Carleo, Real-time quantum dynamics of thermal states with neural thermofields, Phys. Rev. B 109, 235120 (2024).
- K. Donatella, Z. Denis, A. Le Boité, and C. Ciuti, Dynamics with autoregressive neural quantum states: Application to critical quench dynamics, Phys. Rev. A 108, 022210 (2023).
- L. Gravina, V. Savona, and F. Vicentini, Neural projected quantum dynamics: A systematic study, Quantum 9, 1803 (2025).
- A. Van de Walle, M. Schmitt, and A. Bohrdt, Many-body dynamics with explicitly time-dependent neural quantum states, Mach. Learn.: Sci. Technol. 6, 045011 (2025).
- L. Mauron and G. Carleo, Challenging the quantum advantage frontier with large-scale classical simulations of annealing dynamics, arXiv:2503.08247.
- O. Sharir, Y. Levine, N. Wies, G. Carleo, and A. Shashua, Deep autoregressive models for the efficient variational simulation of many-body quantum systems, Phys. Rev. Lett. 124, 020503 (2020).
- D. Wu, L. Wang, and P. Zhang, Solving statistical mechanics using variational autoregressive networks, Phys. Rev. Lett. 122, 080602 (2019).
- T. D. Barrett, A. Malyshev, and A. I. Lvovsky, Autoregressive neural-network wavefunctions for ab initio quantum chemistry, Nat. Mach. Intell. 4, 351 (2022).
- C. Roth, Iterative retraining of quantum spin models using recurrent neural networks, arXiv:2003.06228.
- M. Hibat-Allah, R. G. Melko, and J. Carrasquilla, Supplementing recurrent neural network wave functions with symmetry and annealing to improve accuracy, arXiv:2207.14314.
- M. Hibat-Allah, E. M. Inack, R. Wiersema, R. G. Melko, and J. Carrasquilla, Variational neural annealing, Nat. Mach. Intell. 3, 952 (2021).
- M. S. Moss, R. Wiersema, M. Hibat-Allah, J. Carrasquilla, and R. G. Melko, Leveraging recurrence in neural network wavefunctions for large-scale simulations of Heisenberg antiferromagnets on the square lattice, Phys. Rev. B 112, 134450 (2025).
- M. S. Moss, R. Wiersema, M. Hibat-Allah, J. Carrasquilla, and R. G. Melko, Leveraging recurrence in neural network wavefunctions for large-scale simulations of Heisenberg antiferromagnets on the triangular lattice, Phys. Rev. B 112, 134449 (2025).
- E. Ibarra-García-Padilla, H. Lange, R. G. Melko, R. T. Scalettar, J. Carrasquilla, A. Bohrdt, and E. Khatami, Autoregressive neural quantum states of Fermi Hubbard models, Phys. Rev. Res. 7, 013122 (2025).
- K. Sprague and S. Czischek, Variational Monte Carlo with large patched transformers, Commun. Phys. 7, 90 (2024).
- D. Luo, Z. Chen, J. Carrasquilla, and B. K. Clark, Autoregressive neural network for simulating open quantum systems via a probabilistic formulation, Phys. Rev. Lett. 128, 090501 (2022).
- D. Luo, Z. Chen, K. Hu, Z. Zhao, V. M. Hur, and B. K. Clark, Gauge-invariant and anyonic-symmetric autoregressive neural network for quantum lattice models, Phys. Rev. Res. 5, 013216 (2023).
- Y.-H. Zhang and M. Di Ventra, Transformer quantum state: A multipurpose model for quantum many-body problems, Phys. Rev. B 107, 075147 (2023).
- H. Lange, G. Bornet, G. Emperauger, C. Chen, T. Lahaye, S. Kienle, A. Browaeys, and A. Bohrdt, Transformer neural networks and quantum simulators: A hybrid approach for simulating strongly correlated systems, Quantum 9, 1675 (2025).
- D. Fitzek, Y. H. Teoh, H. P. Fung, G. A. Dagnew, E. Merali, M. S. Moss, B. MacLellan, and R. G. Melko, RydbergGPT, Mach. Learn.: Sci. Technol. 6, 045057 (2025).
- M. Bortone, Y. Rath, and G. H. Booth, Impact of conditional modelling for a universal autoregressive quantum state, Quantum 8, 1245 (2024).
- A. D. King et al., Beyond-classical computation in quantum simulation, Science 388, 199 (2025).
- D.-L. Deng, X. Li, and S. Das Sarma, Quantum entanglement in neural network states, Phys. Rev. X 7, 021021 (2017).
- X.-Q. Sun, T. Nebabu, X. Han, M. O. Flynn, and X.-L. Qi, Entanglement features of random neural network quantum states, Phys. Rev. B 106, 115138 (2022).
- S. D. Geraedts, R. Nandkishore, and N. Regnault, Many-body localization and thermalization: Insights from the entanglement spectrum, Phys. Rev. B 93, 174202 (2016).
- S. Bravyi, D. DiVincenzo, R. Oliveira, and B. Terhal, The complexity of stoquastic local Hamiltonian problems, Quantum Inf. Comput. 8, 361 (2006).
- K. Hornik, M. Stinchcombe, and H. White, Multilayer feedforward networks are universal approximators, Neural Networks 2, 359 (1989).
- M. Raghu, B. Poole, J. Kleinberg, S. Ganguli, and J. Sohl-Dickstein, On the expressive power of deep neural networks, in Proceedings of the 34th International Conference on Machine Learning (PMLR, Cambridge, MA, 2017), Vol. 70, pp. 2847–2854.
- S. Roca-Jerat, M. Gallego, F. Luis, J. Carrete, and D. Zueco, Transformer wave function for quantum long-range models, Phys. Rev. B 110, 205147 (2024).
- L. L. Viteritti, R. Rende, and F. Becca, Transformer variational wave functions for frustrated quantum spin systems, Phys. Rev. Lett. 130, 236401 (2023).
- R. Rende and L. L. Viteritti, Are queries and keys always relevant? A case study on transformer wave functions, Mach. Learn.: Sci. Technol. 6, 010501 (2025).
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, Attention is all you need, in Advances in Neural Information Processing Systems (Curran Associates, Inc., Red Hook, NY, 2017), Vol. 30.
- G. Ke, D. He, and T.-Y. Liu, Rethinking positional encoding in language pre-training, in 9th International Conference on Learning Representations, Vienna, Austria (OpenReview.net, 2021).
- U. Wennberg and G. E. Henter, The case for translation-invariant self-attention in transformer-based language models, in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2 Short Papers) (Association for Computational Linguistics, 2021), pp. 130–140.
- P. Shaw, J. Uszkoreit, and A. Vaswani, Self-attention with relative position representations, in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics Human Language Technologies (Volume 2 Short Papers) (Association for Computational Linguistics, New Orleans, Louisiana, 2018), pp. 464–468.
- J. L. Ba, J. R. Kiros, and G. E. Hinton, Layer normalization, arXiv:1607.06450.
- R. M. Neal, Connectionist learning of belief networks, Artif. Intell. 56, 71 (1992).
- K. He, X. Zhang, S. Ren, and J. Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, in Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV) (IEEE Computer Society, Los Alamitos, CA, 2015), pp. 1026–1034.
- A. Daniely, R. Frostig, and Y. Singer, Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity, in Advances in Neural Information Processing Systems (Curran Associates, Inc., Red Hook, NY, 2016), Vol. 29, pp. 2261–2269.
- M. B. Hastings, I. González, A. B. Kallin, and R. G. Melko, Measuring Rényi entanglement entropy in quantum Monte Carlo simulations, Phys. Rev. Lett. 104, 157201 (2010).
- M. E. J. Newman and G. T. Barkema, Monte Carlo Methods in Statistical Physics (Oxford University Press, Oxford, UK, 1999).
- T. Grover and M. P. A. Fisher, Entanglement and the sign structure of quantum states, Phys. Rev. A 92, 042308 (2015).
- T. Grover and M. P. A. Fisher, Quantum disentangled liquids, J. Stat. Mech. (2014) P10010.
- A. Pal and D. A. Huse, Many-body localization phase transition, Phys. Rev. B 82, 174411 (2010).
- D. A. Abanin, E. Altman, I. Bloch, and M. Serbyn, Colloquium: Many-body localization, thermalization, and entanglement, Rev. Mod. Phys. 91, 021001 (2019).
- D. Wu, R. Rossi, F. Vicentini, and G. Carleo, From tensor-network quantum states to tensorial recurrent neural networks, Phys. Rev. Res. 5, L032001 (2023).
- Y. Levine, O. Sharir, and A. Shashua, Benefits of depth for long-term memory of recurrent networks, in 6th International Conference on Learning Representations, Vancouver, BC, Canada (OpenReview.net, 2018).
- Y. Levine and. O. Sharir, N. Cohen, and A. Shashua, Quantum entanglement in deep learning architectures, Phys. Rev. Lett. 122, 065301 (2019).
- T.-H. Yang, M. Soleimanifar, T. Bergamaschi, and J. Preskill, When can classical neural networks represent quantum states? arXiv:2410.23152.
- H.-Q. Zhou, J.-H. Zhao, and B. Li, Fidelity approach to quantum phase transitions: Finite-size scaling for the quantum ising model in a transverse field, J. Phys. A: Math. Theor. 41, 492002 (2008).
- M. Campostrini, J. Nespolo, A. Pelissetto, and E. Vicari, Finite-size scaling at first-order quantum transitions, Phys. Rev. Lett. 113, 070402 (2014).
- H. Tian, T. He, and X. Wu, Finite-size scaling of half-chain entanglement entropy in the one-dimensional transverse field ising model and the model, Phys. Rev. B 111, 104437 (2025).
- Y. Y. Atas, E. Bogomolny, O. Giraud, and G. Roux, Distribution of the ratio of consecutive level spacings in random matrix ensembles, Phys. Rev. Lett. 110, 084101 (2013).
- V. A. Marchenko and L. A. Pastur, Distribution of eigenvalues for some sets of random matrices, Math. USSR-Sb. 1, 457 (1967).
- M. V. Berry and M. Tabor, Level clustering in the regular spectrum, Proc. R. Soc. London, Ser. A 356, 375 (1977).
- O. Bohigas, M. J. Giannoni, and C. Schmit, Characterization of chaotic quantum spectra and universality of level fluctuation laws, Phys. Rev. Lett. 52, 1 (1984).
- T. A. Brody, J. Flores, J. B. French, P. A. Mello, A. Pandey, and S. S. M. Wong, Random-matrix physics: Spectrum and strength fluctuations, Rev. Mod. Phys. 53, 385 (1981).
- T. Guhr, A. Müller-Groeling, and H. A. Weidenmüller, Random-matrix theories in quantum physics: Common concepts, Phys. Rep. 299, 189 (1998).
- P. J. Forrester, Log-gases and random matrices, London Mathematical Society Monographs (Princeton University Press, Princeton, NJ, 2010), Vol. 34.
- The Oxford handbook of random matrix theory, edited by G. Akemann, J. Baik, and P. Di Francesco (Oxford University Press, Oxford, UK, 2011).
- M. Rigol, V. Dunjko, V. Yurovsky, and M. Olshanii, Relaxation in a completely integrable many-body quantum system: An ab initio study of the dynamics of the highly excited states of 1D lattice hard-core bosons, Phys. Rev. Lett. 98, 050405 (2007).
- L. Vidmar and M. Rigol, Generalized Gibbs ensemble in integrable lattice models, J. Stat. Mech. (2016) 064007.
- K. Cho, B. van Merriënboer, D. Bahdanau, and Y. Bengio, On the properties of neural machine translation: Encoder–decoder approaches, in Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation (Association for Computational Linguistics, Doha, Qatar, 2014), pp. 103–111.
- X. Glorot and Y. Bengio, Understanding the difficulty of training deep feedforward neural networks, in Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, edited by Y. W. Teh and M. Titterington (PMLR, Chia Laguna Resort, Sardinia, Italy, 2010), Vol. 9, pp. 249–256.
- Z.-C. Yang, C. Chamon, A. Hamma, and E. R. Mucciolo, Two-component structure in the entanglement spectrum of highly excited states, Phys. Rev. Lett. 115, 267206 (2015).
- D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, arXiv:1412.6980.
- K. P. Murphy, Machine Learning: A Probabilistic Perspective (MIT Press, Cambridge, MA, 2012).
- P. Zhang, H. Shen, and H. Zhai, Machine learning topological invariants with neural networks, Phys. Rev. Lett. 120, 066401 (2018).
- W. T. Lou, H. Sutterud, G. Cassella, W. M. C. Foulkes, J. Knolle, D. Pfau, and J. S. Spencer, Neural wave functions for superfluids, Phys. Rev. X 14, 021030 (2024).
- Z. Qin, W. Sun, H. Deng, D. Li, Y. Wei, B. Lv, J. Yan, L. Kong, and Y. Zhong, cosFormer: Rethinking softmax in attention, in The Tenth International Conference on Learning Representations (OpenReview.net, 2022).
- J. Zhu, X. Chen, K. He, Y. LeCun, and Z. Liu, Transformers without normalization, arXiv:2503.10622.
- P. Calabrese and J. Cardy, Entanglement entropy and quantum field theory, J. Stat. Mech. (2004) P06002.
- F. D. M. Haldane, Exact Jastrow-Gutzwiller resonating-valence-bond ground state of the spin- antiferromagnetic Heisenberg chain with 1/ exchange, Phys. Rev. Lett. 60, 635 (1988).
- B. S. Shastry, Exact solution of an Heisenberg antiferromagnetic chain with long-ranged interactions, Phys. Rev. Lett. 60, 639 (1988).
- D. Han, Y. Pu, Z. Xia, Y. Han, X. Pan, X. Li, J. Lu, S. Song, and G. Huang, Bridging the divide: Reconsidering softmax and linear attention, in Advances in Neural Information Processing Systems (Curran Associates, Inc., Red Hook, NY, 2024), Vol. 37, pp. 79221–79245.
- S. Fort and A. Scherlis, The goldilocks zone: Towards better understanding of neural network loss landscapes, in Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19) (AAAI Press, 2019), pp. 3574–3581.
- A. Vysogorets, A. Dawid, and J. Kempe, Deconstructing the goldilocks zone of neural network initialization, in Proceedings of the 41st International Conference on Machine Learning (ICML) (PMLR, Vienna, Austria, 2024), Vol. 235, pp. 49717–49732.
- S. Sorella and L. Capriotti, Green function Monte Carlo with stochastic reconfiguration: An effective remedy for the sign problem, Phys. Rev. B 61, 2599 (2000).
- S. Sorella, M. Casula, and D. Rocca, Weak binding between two aromatic rings: Feeling the van der Waals attraction by quantum Monte Carlo methods, J. Chem. Phys. 127, 014105 (2007),.
- G. Carleo, K. Choo, D. Hofmann, J. ET Smith, T. Westerhout, F. Alet, E. J. Davis, S. Efthymiou, I. Glasser, S.-H. Lin, M. Mauri, G. Mazzola, C. B. Pereira, and F. Vicentini, Netket: A machine learning toolkit for many-body quantum systems, SoftwareX 10, 100311 (2019).
- F. Vicentini, D. Hofmann, A. Szabó, D. Wu, C. Roth, C. Giuliani, G. Pescia, J. Nys, V. Vargas-Calderón, N. Astrakhantsev, and G. Carleo, NetKet 3: Machine learning toolbox for many-body quantum systems, SciPost Phys. Codebases 7 (2022).
- J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang, JAX: Composable transformations of Python+NumPy programs (2018), https://github.com/jax-ml/jax.
- J. Heek, A. Levskaya, A. Oliver, M. Ritter, B. Rondepierre, A. Steiner, and M. van Zee, Flax: A neural network library and ecosystem for JAX (2024), https://github.com/google/flax.
- https://github.com/andrewjreissaty91/RandomRNN_Entanglement_VMC.
- www.vectorinstitute.ai/#partners.
- https://github.com/andrewjreissaty91/RandomATF_Entanglement_VMC.
- J. V. Uspensky, Introduction to Mathematical Probability (McGraw-Hill, New York, 1937).
- H. Fischer, A history of the central limit theorem: From classical to modern probability theory, Sources and Studies in the History of Mathematics and Physical Sciences (Springer New York, New York, NY, 2011), 1st ed.