- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Phase transition on a context-sensitive random language model with short-range interactions

Yuma Toji1, Jun Takahashi2, Vwani Roychowdhury3, and Hideyuki Miyahara1,4,*

  • 1Division of Information Science and Technology, Graduate School of Information Science and Technology, Hokkaido University, Sapporo, Hokkaido 060-0814, Japan
  • 2Institute for Solid State Physics, The University of Tokyo, 5-1-5 Kashiwanoha, Kashiwa, Chiba 277-8581, Japan
  • 3Department of Electrical and Computer Engineering, Henry Samueli School of Engineering and Applied Science, University of California, Los Angeles, California 90095, USA
  • 4School of Computing, Institute of Science Tokyo, Ookayama, Tokyo 152-8550, Japan

  • *Contact author: miyahara@ist.hokudai.ac.jp; hmiyahara512@gmail.com
PDF

Phys. Rev. Research 8, 043073 – Published 19 October, 2026

DOI: https://doi.org/10.1103/vnf1-mfrs

Abstract

Since the random language model was proposed by De Giuli [Phys. Rev. Lett. 122, 128301 (2019)], language models have been investigated intensively from the viewpoint of statistical mechanics. Recently, the existence of a Berezinskii-Kosterlitz-Thouless transition was numerically reported in models with long-range interactions between symbols. In statistical mechanics, it has long been known that long-range interactions can induce phase transitions. Therefore, it has remained unclear whether phase transitions observed in language models originate from genuinely linguistic properties that are absent in conventional spin models. In this study, we construct a random language model with short-range interactions and numerically investigate its statistical properties. Our model belongs to the class of context-sensitive grammars in the Chomsky hierarchy and allows explicit reference to contexts. We find that a phase transition occurs even when the model refers only to contexts whose length remains constant with respect to the sentence length. This result indicates that finite-temperature phase transitions in language models are genuinely induced by the intrinsic nature of language, rather than by long-range interactions.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (35)

  1. K. Huang, Statistical Mechanics, 2nd ed. (Wiley, New York, 1987).
  2. N. Goldenfeld, Lectures on Phase Transitions and the Renormalization Group (Addison-Wesley, Reading, MA, 1992).
  3. L. D. Landau and E. M. Lifshitz, Statistical Physics, Part 1, 3rd ed. (Pergamon Press, Oxford, 1980).
  4. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, Scaling laws for neural language models, arXiv:2001.08361.
  5. J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al., Emergent abilities of large language models, arXiv:2206.07682.
  6. E. De Giuli, Random language model, Phys. Rev. Lett. 122, 128301 (2019).
  7. K. Nakaishi and K. Hukushima, Absence of phase transition in random language model, Phys. Rev. Res. 4, 023156 (2022).
  8. F. Lalegani and E. De Giuli, Robustness of the random language model, Phys. Rev. E 109, 054313 (2024).
  9. Y. Toji, J. Takahashi, V. Roychowdhury, and H. Miyahara, Berezinskii-Kosterlitz-Thouless transition in a context-sensitive random language model, Phys. Rev. E 113, 015305 (2026).
  10. V. Berezinskii, Destruction of long-range order in one-dimensional and two-dimensional systems having a continuous symmetry group. I. Classical systems, Sov. Phys. JETP 32, 493 (1971).
  11. V. Berezinskii, Destruction of long-range order in one-dimensional and two-dimensional systems possessing a continuous symmetry group. II. Quantum systems, Sov. Phys. JETP 34, 610 (1972).
  12. J. M. Kosterlitz and D. J. Thouless, Ordering, metastability and phase transitions in two-dimensional systems, J. Phys. C 6, 1181 (1973).
  13. J. M. Kosterlitz, Kosterlitz–Thouless physics: A review of key issues, Rep. Prog. Phys. 79, 026001 (2016).
  14. J. M. Kosterlitz, The critical properties of the two-dimensional model, J. Phys. C 7, 1046 (1974).
  15. F. J. Dyson, Existence of a phase-transition in a one-dimensional Ising ferromagnet, Commun. math. Phys. 12, 91 (1969).
  16. F. J. Dyson, Non-existence of spontaneous magnetization in a one-dimensional Ising ferromagnet, Matematika 16, 113 (1972).
  17. F. J. Dyson, An Ising ferromagnet with discontinuous long-range order, Commun. Math. Phys. 21, 269 (1971).
  18. Y. Tomita, Monte Carlo study of one-dimensional Ising models with long-range interactions, J. Phys. Soc. Jpn. 78, 014002 (2009).
  19. J. Martínez-Herrera, O. Rodríguez-López, and M. Solís, Critical temperature of one-dimensional Ising model with long-range interaction revisited, Physica A 596, 127136 (2022).
  20. F. Y. Wu, The Potts model, Rev. Mod. Phys. 54, 235 (1982).
  21. M. A. Harrison, Introduction to Formal Language Theory (Addison-Wesley, Reading, MA, 1978).
  22. G. Rozenberg and A. Salomaa, Handbook of Formal Languages, Vol. 3: Beyond Words (Springer Science & Business Media,New York, 2012).
  23. P. Linz and S. H. Rodger, An Introduction to Formal Languages and Automata (Jones & Bartlett Learning, Burlington, MA, 2022).
  24. A. Meduna, Formal Languages and Computation (Auerbach Publishers, Incorporated, Boca Raton, FL, 2014).
  25. D. Landau and K. Binder, A Guide to Monte Carlo Simulations in Statistical Physics (Cambridge University Press, Cambridge, 2021).
  26. J. Binney, N. Dowrick, A. Fisher, and M. Newman, The Theory of Critical Phenomena: An Introduction to the Renormalization Group (Clarendon Press, Oxford, England, 1992).
  27. M. Newman and G. Barkema, Monte Carlo Methods in Statistical Physics (Clarendon Press, Oxford, England, 1999).
  28. G. Jäger and J. Rogers, Formal language theory: Refining the Chomsky hierarchy, Philos. Trans. R. Soc. B 367, 1956 (2012).
  29. L. M. Tuan, T. T. Long, D. X. Nui, P. T. Minh, N. D. Trung Kien, and D. X. Viet, Binder ratio in the two-dimensional q-state clock model, Phys. Rev. E 106, 034138 (2022).
  30. M. Hasenbusch, The Binder cumulant at the Kosterlitz–Thouless transition, J. Stat. Mech. (2008) P08003.
  31. H. W. Lin and M. Tegmark, Critical behavior in physics and probabilistic formal languages, Entropy 19, 299 (2017).
  32. V. Ardourel and S. Bangu, Finite-size scaling theory: Quantitative and qualitative approaches to critical phenomena, Stud. Hist. Philos. Sci. 100, 99 (2023).
  33. Y.-D. Hsieh, Y.-J. Kao, and A. W. Sandvik, Finite-size scaling method for the Berezinskii–Kosterlitz–Thouless transition, J. Stat. Mech. (2013) P09001.
  34. Z. Zuo, S. Yin, X. Cao, and F. Zhong, Scaling theory of the Kosterlitz-Thouless phase transition, Phys. Rev. B 104, 214108 (2021).
  35. H. Ueda, K. Okunishi, K. Harada, R. Krčmár, A. Gendiar, S. Yunoki, and T. Nishino, Finite-m scaling analysis of Berezinskii-Kosterlitz-Thouless phase transitions and entanglement spectrum for the six-state clock model, Phys. Rev. E 101, 062111 (2020).

Outline

Information

Phys. Rev. Research 8, 043073– Published 19 October, 2026
  • Received 31 March 2026
  • Accepted 21 July 2026
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation