- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Entropy and type-token ratio in gigaword corpora

Pablo Rosillo-Rodes*, Maxi San Miguel, and David Sánchez

  • *Contact author: prosillo@ifisc.uib-csic.es
PDF

Phys. Rev. Research 7, 033054 – Published 14 July, 2025

DOI: https://doi.org/10.1103/rxxz-lk3n

Abstract

There are different ways of measuring diversity in complex systems. In particular, in language, lexical diversity is characterized in terms of the type-token ratio and the word entropy. We here investigate both diversity metrics in six massive linguistic data sets in English, Spanish, and Turkish, consisting of books, news articles, and tweets. These gigaword corpora correspond to languages with distinct morphological features and differ in registers and genres, thus constituting a varied testbed for a quantitative approach to lexical diversity. We unveil an empirical functional relation between entropy and type-token ratio of texts of a given corpus and language, which is a consequence of the statistical laws observed in natural language. Further, in the limit of large text lengths we find an analytical expression for this relation relying on both Zipf and Heaps laws that agrees with our empirical findings.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (101)

  1. W. Johnson, Studies in language behavior: A program of research, Phychol. Monogr. 56, 1 (1944).
  2. M. C. Templin, Certain Language Skills in Children: Their Development and Interrelationships (University of Minnesota Press, Minneapolis, MN, 1957), Vol. 26.
  3. A. Hardie and T. McEnery, Statistics, in Encyclopedia of Language and Linguistics, 2nd ed., edited by K. Brown (Elsevier, Amsterdam, 2006).
  4. K. Kettunen, Can type-token ratio be used to show morphological complexity of languages? J. Quant. Linguist. 21, 223 (2014).
  5. N. Bermel, Corpora and quantitative data in Slavic languages, Russ. Linguist. 39, 275 (2015).
  6. T. Litvinova, P. Seredin, O. Litvinova, and O. Zagorovskaya, Differences in type-token ratio and part-of-speech frequencies in male and female Russian written texts, in Proceedings of the Workshop on Stylistic Variation, edited by J. Brooke, T. Solorio, and M. Koppel (Association for Computational Linguistics, Copenhagen, Denmark, 2017), pp. 69–73.
  7. C. W. Hess, K. M. Sefton, and R. G. Landry, Sample size and type-token ratios for oral language of preschool children, J. Speech Lang. Hear Res. 29, 129 (1986).
  8. T. C. Manschreck, B. A. Maher, T. M. Hoover, and D. Ames, The type–token ratio in schizophrenic disorders: Clinical and research value, Psychol. Med. 14, 151 (1984).
  9. L. Jost, Entropy and diversity, Oikos 113, 363 (2006).
  10. H. Tuomisto, A consistent terminology for quantifying species diversity? Yes, it does exist, Oecologia 164, 853 (2010).
  11. O. Mazzarisi, A. de Azevedo-Lopes, J. J. Arenzon, and F. Corberi, Maximal diversity and Zipf's law, Phys. Rev. Lett. 127, 128301 (2021).
  12. C. Bentz, D. Alikaniotis, M. Cysouw, and R. Ferrer-i-Cancho, The entropy of words–learnability and expressivity across more than 1000 languages, Entropy 19, 275 (2017).
  13. M. A. Montemurro and D. H. Zanette, Universal entropy of word ordering across linguistic families, PLoS ONE 6, e19875 (2011).
  14. C. E. Shannon, A mathematical theory of communication, Bell Syst. Tech. J. 27, 379 (1948).
  15. T. M. Cover and J. A. Thomas, Elements of Information Theory (Wiley, New York, 1999).
  16. K. Liu, R. Ye, L. Zhongzhu, and R. Ye, Entropy-based discrimination between translated Chinese and original Chinese using data mining techniques, PLoS ONE 17, e0265633 (2022).
  17. R. Friedrich, M. Luzzatto, and E. Ash, Entropy in legal language, in NLLP 2020 Natural Legal Language Processing Workshop 2020. Proceedings of the Natural Legal Language Processing Workshop 2020 co-located with the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD 2020) (CEUR-WS, Aachen, Germany, 2020), Vol. 2645, pp. 25–30.
  18. M. Gerlach, F. Font-Clos, and E. G. Altmann, Similarity of symbol frequency distributions with heavy tails, Phys. Rev. X 6, 021009 (2016).
  19. M. Gerlach, H. Shi, and L. A. N. Amaral, A universal information theoretic approach to the identification of stopwords, Nat. Mach. Intell. 1, 606 (2019).
  20. M. Mohseni, C. Redies, and V. Gast, Comparative analysis of preference in contemporary and earlier texts using entropy measures, Entropy 25, 486 (2023).
  21. I. I. Eliazar and I. M. Sokolov, Diversity of Poissonian populations, Phys. Rev. E 81, 011122 (2010).
  22. J. R. Romero-Arias, G. Ramírez-Santiago, J. X. Velasco-Hernández, L. Ohm, and M. Hernández-Rosales, Model for breast cancer diversity and spatial heterogeneity, Phys. Rev. E 98, 032401 (2018).
  23. G. K. Zipf, The psychology of language, in Encyclopedia of Psychology (Philosophical Library, New York, 1946), pp. 332–341.
  24. G. Herdan, Type-Token Mathematics: A Textbook of Mathematical Linguistics (Moulton & Co, Irvine, CA, 1960).
  25. H. Heaps, Information Retrieval: Computational and Theoretical Aspects (Academic Press, New York, 1978).
  26. E. G. Altmann and M. Gerlach, Statistical Laws in Linguistics, in Creativity and Universality in Language, edited by M. Degli Esposti, E. G. Altmann, and F. Pachet (Springer International Publishing, Cham, 2016), pp. 7–26.
  27. T. Stanisz, S. Drożdż, and J. Kwapień, Complex systems approach to natural language, Phys. Rep. 1053, 1 (2024).
  28. I. Arnon, S. Kirby, J. A. Allen, C. Garrigue, E. L. Carroll, and E. C. Garland, Whale song shows language-like statistical structure, Science 387, 649 (2025).
  29. M. Mitzenmacher, A brief history of generative models for power law and lognormal distributions, Internet Math. 1, 226 (2004).
  30. M. Á. Serrano, A. Flammini, and F. Menczer, Modeling statistical properties of written text, PLoS ONE 4, e5372 (2009).
  31. M. Gerlach and E. G. Altmann, Stochastic model for the vocabulary growth in natural languages, Phys. Rev. X 3, 021006 (2013).
  32. D. Malvern, B. Richards, N. Chipere, and P. Durán, Lexical Diversity and Language Development (Springer, New York, 2004).
  33. Y. Shi and L. Lei, Lexical richness and text length: An entropy-based perspective, J. Quant. Linguist. 29, 62 (2022).
  34. C. Gregori-Signes and B. Clavel-Arroitia, Analysing lexical density and lexical diversity in university students' written discourse, Procedia Soc. Behav. Sci. 198, 546 (2015).
  35. A. Koplenig, S. Wolfer, and C. Müller-Spitzer, Studying lexical dynamics and language change via generalized entropies: The problem of sample size, Entropy 21, 464 (2019).
  36. A. E. Magurran, Measuring biological diversity, Curr. Biol. 31, R1174 (2021).
  37. B. Comrie, Language Universals and Linguistic Typology: Syntax and Morphology (University of Chicago Press, Chicago, IL, 1989).
  38. J. R. Taylor, The Oxford Handbook of the Word (OUP, Oxford, 2015).
  39. S. Bird, Nltk: The natural language toolkit, in Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions (Association for Computational Linguistics, Sydney, Australia, 2006), pp. 69–72.
  40. M. Gerlach and F. Font-Clos, A standardized project gutenberg corpus for statistical analysis of natural language and quantitative linguistics, Entropy 22, 126 (2020).
  41. GitHub repository with codes and access to corpora, https://github.com/pablorosillo/entropy_ttr_gigaword/.
  42. M. Hart, Project Gutenberg (University of Illinois, Urbana, 1971), https://www.gutenberg.org.
  43. J. Dunn, Natural Language Processing for Corpus Linguistics (Cambridge University Press, Cambridge, 2022).
  44. L. Langer, M. Burghardt, R. Borgards, K. Böhning-Gaese, R. Seppelt, and C. Wirth, The rise and fall of biodiversity in literature: A comprehensive quantification of historical changes in the use of vernacular labels for biological taxa in Western creative literature, People Nat. 3, 1093 (2021).
  45. Q. He, S. Cheng, Z. Li, R. Xie, and Y. Xiao, Can pre-trained language models interpret similes as smart as human? arXiv:2203.08452.
  46. B. Shade and E. G. Altmann, Quantifying the dissimilarity of texts, Information 14, 271 (2023).
  47. A. Corral and I. Serra, The brevity law as a scaling law, and a possible origin of Zipf's law for word frequencies, Entropy 22, 224 (2020).
  48. M. Davies, Corpus del Español: Web/Dialects (Brigham Young University, Provo, Utah, 2016), https://www.corpusdelespanol.org/web-dial/.
  49. D. Biber, M. Davies, J. K. Jones, and N. Tracy-Ventura, Spoken and written register variation in Spanish: A multidimensional analysis, Corpora 1, 1 (2006).
  50. D. Daidone, Preterite and Imperfect in Spanish instructor oral input and spanish language corpora, Hispania 102, 45 (2019).
  51. Common Crawl, Common Crawl Overview (Common Crawl Foundation, San Francisco, California, 2024), https://commoncrawl.org/overview.
  52. A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, Unsupervised cross-lingual representation learning at scale, in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, edited by D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (Association for Computational Linguistics, Online, 2020), pp. 8440–8451.
  53. G. Wenzek, M.-A. Lachaux, A. Conneau, V. Chaudhary, F. Guzmán, A. Joulin, and E. Grave, CCNet: Extracting high quality monolingual data sets from web crawl data, in Proceedings of the Twelfth Language Resources and Evaluation Conference, edited by N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, and S. Piperidis (European Language Resources Association, Marseille, France, 2020).
  54. For our corpora, we will maintain references to Twitter and not to X throughout the manuscript, as Twitter was the name of the social network while the posts compilation was performed.
  55. J. Eisenstein, B. O'Connor, N. A. Smith, and E. P. Xing, Diffusion of lexical change in social media, PLoS ONE 9, e113114 (2014).
  56. B. Gonçalves and D. Sánchez, Crowdsourcing dialect characterization through Twitter, PLoS ONE 9, e112074 (2014).
  57. B. Gonçalves, L. Loureiro-Porto, J. J. Ramasco, and D. Sánchez, Mapping the americanization of English in space and time, PLoS ONE 13, e0197741 (2018).
  58. J. L. Abitbol, M. Karsai, J.-P. Magué, J.-P. Chevrot, and E. Fleury, Socioeconomic dependences of linguistic patterns in Twitter: A multivariate analysis, in Proceedings of the 2018 World Wide Web Conference, WWW '18 (International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, 2018), pp. 1125–1134.
  59. J. Grieve, C. Montgomery, A. Nini, A. Murakami, and D. Guo, Mapping lexical dialect variation in British English using Twitter, Front. Artif. Intell. 2, 11 (2019).
  60. T. Alshaabi, J. L. Adams, M. V. Arnold, J. R. Minot, D. R. Dewhurst, A. J. Reagan, C. M. Danforth, and P. S. Dodds, Storywrangler: A massive exploratorium for sociolinguistic, cultural, socioeconomic, and political timelines using Twitter, Sci. Adv. 7, eabe6534 (2021).
  61. T. Louf, D. Sánchez, and J. J. Ramasco, Capturing the diversity of multilingual societies, Phys. Rev. Res. 3, 043146 (2021).
  62. E. S. Tellez, D. Moctezuma, S. Miranda, M. Graff, and G. Ruiz, Regionalized models for Spanish language variations based on Twitter, Lang. Resour. Eval. 57, 1697 (2023).
  63. T. Louf, J. J. Ramasco, D. Sánchez, and M. Karsai, When dialects collide: How socioeconomic mixing affects language use, arXiv:2307.10016.
  64. J. Dunn, Syntactic variation across the grammar: modelling a complex adaptive system, Front. Complex Syst. 1, 1273741 (2023).
  65. R. Ghosh, T. Surachawala, and K. Lerman, Entropy-based classification of ‘retweeting’ activity on Twitter, arXiv:1106.0346.
  66. J. Paryani, A. K. TK, and K. George, Entropy-based model for estimating veracity of topics from tweets, in Computational Collective Intelligence: 9th International Conference, ICCCI 2017, Nicosia, Cyprus, September 27-29, 2017, Proceedings, Part II 9 (Springer, New York, 2017), pp. 417–427.
  67. A. Kanavos, G. Vonitsanos, A. Mohasseb, and P. Mylonas, An entropy-based evaluation for sentiment analysis of stock market prices using twitter data, in 2020 15th International Workshop on Semantic and Social Media Adaptation and Personalization (SMA) (IEEE, Piscataway, New Jersey, 2020), pp. 1–7.
  68. Andrew Hayden and J. Riesa, Compact Language Detector 2 (CLD2), https://github.com/CLD2Owners/cld2.
  69. D. C. Van Leijenhorst and T. P. van der Weide, A formal derivation of Heaps' Law, Info. Sci. 170, 263 (2005).
  70. L. Lü, Z.-K. Zhang, and T. Zhou, Zipf's law leads to heaps' law: Analyzing their relation in finite-size systems, PloS One 5, 1 (2010).
  71. I. I. Eliazar and M. H. Cohen, Power-law connections: From Zipf to Heaps and beyond, Ann. Phys. 332, 56 (2012).
  72. F. Font-Clos, G. Boleda, and Á. Corral, A scaling law beyond Zipf's law and its relation to Heaps' law, New J. Phys. 15, 093033 (2013).
  73. V. Loreto, V. D. Servedio, S. H. Strogatz, and F. Tria, Dynamics on expanding spaces: Modeling the emergence of novelties, in Creativity and Universality in Language (Springer, New York, 2016), pp. 59–83.
  74. M. A. Serrano, A. Flammini, and F. Menczer, Modeling statistical properties of written text, PLoS ONE 4, e1 (2009).
  75. A. M. Petersen, J. N. Tenenbaum, S. Havlin, H. E. Stanley, and M. Perc, Languages cool as they expand: Allometric scaling and the decreasing need for new words, Sci. Rep. 2, 943 (2012).
  76. T. Yamamoto, S. Yamada, and T. Mizuguchi, Negative correlation of word rank sequence in written texts, Eur. Phys. J. B 94, 200 (2021).
  77. R. F. i Cancho and R. V. Solé, Two regimes in the frequency of words and the origins of complex lexicons: Zipf's law revisited, J. Quant. Linguist. 8, 165 (2001).
  78. V. V. Bochkarev, E. Y. Lerner, and A. V. Shevlyakova, Deviations in the Zipf and Heaps laws in natural languages, in 2nd International Conference on Mathematical Modeling in Physical Sciences 2013 (IC-MSQUARE 2013), Journal of Physics: Conference Series Vol. 490 (IOP Publishing, London, 2014), p. 012009.
  79. T. M. Apostol, An elementary view of Euler's summation formula, Am. Math. Monthly 106, 409 (1999).
  80. J. Liang and J. Todd, The stieltjes constants, J. Res. Nat. Bur. Standards Sect. B 76, 161 (1972).
  81. M. W. Coffey, Series representations for the Stieltjes constants, Rocky Mt. J. Math. 44, 443 (2014).
  82. D. S. Jones, Elementary Information Theory (Oxford University Press, Oxford, 1979).
  83. B. Corominas-Murtra and R. V. Solé, Universality of Zipf's law, Phys. Rev. E 82, 011102 (2010).
  84. M. Visser, Zipf's law, power laws and maximum entropy, New J. Phys. 15, 043021 (2013).
  85. K. W. Vugrin, L. P. Swiler, R. M. Roberts, N. J. Stucky-Mack, and S. P. Sullivan, Confidence region estimation techniques for nonlinear regression in groundwater flow: Three case studies, Water Resour. Res. 43, WR004804 (2007).
  86. G. J. Székely, M. L. Rizzo, and N. K. Bakirov, Measuring and testing dependence by correlation of distances, Ann. Stat. 35, 2769 (2007).
  87. S. Ghosh, Distance correlation in Python (GitHub, San Francisco, California, 2014), https://gist.github.com/satra/aa3d19a12b74e9ab7941.
  88. I. Nemenman, F. Shafee, and W. Bialek, Entropy and inference, revisited, NeurIPS 14, 471 (2001).
  89. A. Arora, C. Meister, and R. Cotterell, Estimating the entropy of linguistic distributions, in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics edited by S. Muresan, P. Nakov, and A. Villavicencio (Association for Computational Linguistics, Dublin, Ireland, 2022), pp. 175–195.
  90. J. De Gregorio, D. Sánchez, and R. Toral, Entropy estimators for markovian sequences: A comparative analysis, Entropy 26, 79 (2024).
  91. C. Bentz, T. Ruzsics, A. Koplenig, and T. Samardžić, A comparison between morphological complexity measures: Typological data vs. language corpora, in Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity (CL4LC), edited by D. Brunato, F. Dell'Orletta, G. Venturi, T. François, and P. Blache (The COLING 2016 Organizing Committee, Osaka, Japan, 2016), pp. 142–153.
  92. W. Ebeling and T. Pöschel, Entropy and long-range correlations in literary english, Europhys. Lett. 26, 241 (1994).
  93. E. G. Altmann, G. Cristadoro, and M. Degli Esposti, On the origin of long-range correlations in texts, Proc. Natl. Acad. Sci. USA 109, 11582 (2012).
  94. M. A. Montemurro and P. A. Pury, Long-range fractal correlations in literary corpora, Fractals 10, 451 (2002).
  95. S. Drożdż, P. Oświe¸cimka, A. Kulig, J. Kwapień, K. Bazarnik, I. Grabska-Gradzińska, J. Rybicki, and M. Stanuszek, Quantifying origin and character of long-range correlations in narrative texts, Info. Sci. 331, 32 (2016).
  96. K. Tanaka-Ishii and A. Bunde, Long-range memory in literary texts: On the universal clustering of the rare words, PloS One 11, e0164658 (2016).
  97. D. Sánchez, L. Zunino, J. De Gregorio, R. Toral, and C. Mirasso, Ordinal analysis of lexical patterns, Chaos 33, 033121 (2023).
  98. A. Kulig, J. Kwapień, T. Stanisz, and S. Drożdż, In narrative texts punctuation marks obey the same statistics as words, Inf. Sci. 375, 98 (2017).
  99. J. Camacho and R. V. Solé, Scaling in ecological size spectra, Europhys. Lett. 55, 774 (2001).
  100. R. Solé and J. Bascompte, Self-Organization in Complex Ecosystems. (MPB-42), Monographs in Population Biology (Princeton University Press, Princeton, New Jersey, 2006).
  101. E. G. Altmann, L. Dias, and M. Gerlach, Generalized entropies and the similarity of texts, J. Stat. Mech. (2017) 014002.

Outline

Information

Phys. Rev. Research 7, 033054– Published 14 July, 2025
  • Received 26 February 2025
  • Accepted 22 May 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation