- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Letter
  • Open Access

Correlation dimension of natural language in a statistical manifold

Xin Du1,* and Kumiko Tanaka-Ishii2,†

  • 1Waseda Research Institute for Science and Engineering, Waseda University, 3-4-1 Okubo, Shinjuku-ku, Tokyo 169-8555, Japan
  • 2Department of Computer Science and Engineering, School of Fundamental Science and Engineering, Waseda University, 3-4-1 Okubo, Shinjuku-ku, Tokyo 169-8555, Japan

  • *duxin@aoni.waseda.jp
  • †kumiko@waseda.jp
PDF

Phys. Rev. Research 6, L022028 – Published 2 May, 2024

DOI: https://doi.org/10.1103/PhysRevResearch.6.L022028

Abstract

The correlation dimension of natural language is measured by applying the Grassberger-Procaccia algorithm to high-dimensional sequences produced by a large-scale language model. This method, previously studied only in a Euclidean space, is reformulated in a statistical manifold via the Fisher-Rao distance. Language exhibits a multifractal, with global self-similarity and a universal dimension around 6.5, which is smaller than those of simple discrete random sequences and larger than that of a Barabási-Albert process. Long memory is the key to producing self-similarity. Our method is applicable to any probabilistic model of real-world discrete sequences, and we show an application to music data.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (35)

  1. P. Grassberger and I. Procaccia, Characterization of strange attractors, Phys. Rev. Lett. 50, 346 (1983).
  2. A. R. Osborne and A. Provenzale, Finite correlation dimension for stochastic systems with power-law spectra, Physica D 35, 357 (1989).
  3. L. Lacasa and J. Gómez-Gardenes, Correlation dimension of complex networks, Phys. Rev. Lett. 110, 168703 (2013).
  4. W. Li, Mutual information functions of natural language texts (Citeseer, 1989)
  5. E. G. Altmann, G. Cristadoro, and M. D. Esposti, On the origin of long-range correlations in texts, Proc. Natl. Acad. Sci. USA 109, 11582 (2012).
  6. K. Tanaka-Ishii and A. Bunde, Long-range memory in literary texts: On the universal clustering of the rare words, PLoS ONE 11, e0164658 (2016).
  7. I. Doxas, S. Dennis, and W. L. Oliver, The dimensionality of discourse, Proc. Natl. Acad. Sci. USA 107, 4866 (2010).
  8. T. Kobayashi and K. Tanaka-Ishii, Taylor's law for human linguistic sequences, in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistic (2018), p. 1138.
  9. K. Tanaka-Ishii and T. Kobayashi, Taylor's law for linguistic sequences and random walk models, J. Phys. Commun. 2, 115024 (2018).
  10. M. Ausloos, Measuring complexity with multifractals in texts. Translation effects, Chaos, Solitons Fractals 45, 1349 (2012).
  11. A. Radford, J. Wu, R. Child et al., Language models are unsupervised multitask learners, OpenAI blog 1, 9 (2019).
  12. OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman et al., GPT-4 technical report, arXiv:2303.08774.
  13. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., Llama 2: Open foundation and fine-tuned chat models, arXiv:2307.09288.
  14. The Yi model, (2024), visited in January 2024, https://huggingface.co/01-ai/Yi-34B.
  15. C. R. Rao, Information and the accuracy attainable in the estimation of statistical parameters, in Breakthroughs in Statistics: Foundations and Basic Theory (Springer, New York, 1992), pp. 235–247.
  16. S.-I. Amari, Differential-Geometrical Methods in Statistics, Vol. 28 (Springer, New York, 2012).
  17. Y. B. Pesin, On rigorous mathematical definitions of correlation dimension and generalized spectrum for dimensions, J. Stat. Phys. 71, 529 (1993).
  18. D. A. Russell, J. D. Hanson, and E. Ott, Dimension of strange attractors, Phys. Rev. Lett. 45, 1175 (1980).
  19. See Supplemental Material at http://link.aps.org/supplemental/10.1103/PhysRevResearch.6.L022028 for theoretical analysis of our method and extensive evaluation results.
  20. E. G. Altmann, J. B. Pierrehumbert, and A. E. Motter, Beyond word frequency: Bursts, lulls, and scaling in the temporal distributions of words, PLoS ONE 4, e7678 (2009).
  21. H.-O. Peitgen, H. Jürgens, D. Saupe, and M. J. Feigenbaum, Chaos and Fractals: New Frontiers of Science, Vol. 7 (Springer, New York, 1992).
  22. J. M. Marstrand, Some fundamental geometrical properties of plane sets of fractional dimensions, Proc. London Math. Soc. s3-4, 257 (1954).
  23. K. Falconer, Fractal Geometry: Mathematical Foundations and Applications (John Wiley & Sons, England, 2004).
  24. Z. Balogh and A. Iseli, Dimensions of projections of sets on riemannian surfaces of constant curvature, Proc. Am. Math. Soc. 144, 2939 (2016).
  25. https://www.gutenberg.org/ebooks/996.
  26. D. M. Blei, A. Y. Ng, and M. I. Jordan, Latent dirichlet allocation, J. Mach. Learn. Res. 3, 993 (2003).
  27. P. Abry, P. Flandrin, M. S. Taqqu, and D. Veitch, Self-similarity and long-range dependence through the wavelet lens, Theor. Applic. Long-Range Depend. 1, 527 (2003).
  28. D. Kershaw and R. Koeling, Elsevier OA CC-BY corpus, arXiv:2008.00774.
  29. https://plato.stanford.edu/.
  30. J. D. Farmer, Information dimension and the probabilistic structure of chaos, Z. Naturforsch. A 37, 1304 (1982).
  31. A.-L. Barabási and R. Albert, Emergence of scaling in random networks, Science 286, 509 (1999).
  32. R. Rak and E. Rak, The fractional preferential attachment scale-free network model, Entropy 22, 509 (2020).
  33. J. Copet et al., Simple and controllable music generation, in Advances in Neural Information Processing Systems (New Orleans, Louisiana, 2023).
  34. G. Tzanetakis and P. Cook, Musical genre classification of audio signals, IEEE Trans. Speech Audio Process. 10, 293 (2002).
  35. N. Chomsky, Aspects of the Theory of Syntax, Vol. 11 (MIT Press, Cambridge, MA, 2014).

Outline

Information

Phys. Rev. Research 6, L022028– Published 2 May, 2024
  • Received 25 July 2023
  • Revised 24 January 2024
  • Accepted 9 April 2024
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation