- Physical Review Journals
All JournalsPhysics Magazine

PRX Intelligence

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective

Takahiro Sagawa*

  • *Contact author: sagawa@ap.t.u-tokyo.ac.jp
PDF

PRX Intelligence 1, 023013 – Published 5 November, 2026

DOI: https://doi.org/10.1103/tv38-b23y

Abstract

Autoregressive generative models—including Transformers, recurrent neural networks, classical Kalman filters, state space models, and Mamba—all generate sequences by sampling each output from a deterministic summary of the past, producing genuinely non-Markovian observed processes. We develop a general theoretical framework based on stochastic thermodynamics for this class of architectures and introduce the entropy production, which can be efficiently estimated from sampled trajectories without exponential overhead, despite the non-Markovian nature of the observed dynamics. As a proof-of-concept experiment with a large language model (LLM), we evaluate the entropy production for a pretrained Transformer-based model, GPT-2. We find that the token-level entropy production is dominated by a syntactic artifact, while the sentence-level entropy production tends to be larger for causally ordered than for noncausal text sets. This observation is supported by a reevaluation with a substantially larger model, Qwen3-4B-Base. We also demonstrate the framework in the linear Gaussian case, where the model reduces to the Kalman innovation representation and the entropy production admits an analytical expression. We also show that the entropy production decomposes exactly into non-negative per-step contributions in terms of retrospective inference, and each of those terms further splits into information-theoretically meaningful terms: a compression loss and a model mismatch. Our results establish a bridge between stochastic thermodynamics and modern generative models, and provide a starting point for using irreversibility as a quantitative probe of the highly non-Markovian processes generated by models such as LLMs.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (95)

  1. U. Seifert, Stochastic thermodynamics, fluctuation theorems and molecular machines, Rep. Prog. Phys. 75, 126001 (2012).
  2. L. Peliti and S. Pigolotti, Stochastic Thermodynamics: An Introduction (Princeton University Press, Princeton, 2021).
  3. C. Jarzynski, Nonequilibrium equality for free energy differences, Phys. Rev. Lett. 78, 2690 (1997).
  4. G. E. Crooks, Entropy production fluctuation theorem and the nonequilibrium work relation for free energy differences, Phys. Rev. E 60, 2721 (1999).
  5. U. Seifert, Entropy production along a stochastic trajectory and an integral fluctuation theorem, Phys. Rev. Lett. 95, 040602 (2005).
  6. A. Gomez-Marin, J. M. R. Parrondo, and C. Van den Broeck, Lower bounds on dissipation upon coarse graining, Phys. Rev. E 78, 011107 (2008).
  7. É. Roldán and J. M. R. Parrondo, Estimating dissipation from single stationary trajectories, Phys. Rev. Lett. 105, 150607 (2010).
  8. É. Roldán and J. M. R. Parrondo, Entropy production and Kullback–Leibler divergence between stationary trajectories of discrete systems, Phys. Rev. E 85, 031129 (2012).
  9. K. Kawaguchi and Y. Nakayama, Fluctuation theorem for hidden entropy production, Phys. Rev. E 88, 022147 (2013).
  10. M. Kahlen and J. Ehrich, Hidden slow degrees of freedom and fluctuation theorems: An analytically solvable model, J. Stat. Mech. (2018) 063204.
  11. G. E. Crooks and S. E. Still, Marginal and conditional second laws of thermodynamics, Europhys. Lett. 125, 40005 (2019).
  12. D. S. Seara, B. B. Machta, and M. P. Murrell, Irreversibility in dynamical phases and transitions, Nat. Commun. 12, 392 (2021).
  13. T. Speck and U. Seifert, The Jarzynski relation, fluctuation theorems, and stochastic thermodynamics for non-Markovian processes, J. Stat. Mech. (2007) L09002.
  14. T. Ohkuma and T. Ohta, Fluctuation theorems for non-linear generalized Langevin systems, J. Stat. Mech. (2007) P10010.
  15. S. Rahav and C. Jarzynski, Fluctuation relations and coarse-graining, J. Stat. Mech. (2007) P09012.
  16. A. Puglisi, S. Pigolotti, L. Rondoni, and A. Vulpiani, Entropy production and coarse graining in Markov processes, J. Stat. Mech. (2010) P05015.
  17. M. Esposito, Stochastic thermodynamics under coarse graining, Phys. Rev. E 85, 041125 (2012).
  18. T. Munakata and M. L. Rosinberg, Entropy production and fluctuation theorems for Langevin processes under continuous non-Markovian feedback control, Phys. Rev. Lett. 112, 180601 (2014).
  19. M. L. Rosinberg, T. Munakata, and G. Tarjus, Stochastic thermodynamics of Langevin systems under time-delayed feedback control: Second-law-like inequalities, Phys. Rev. E 91, 042114 (2015).
  20. M. Polettini and M. Esposito, Effective thermodynamics for a marginal observer, Phys. Rev. Lett. 119, 240601 (2017).
  21. J. van der Meer, J. Degünther, and U. Seifert, Time-resolved statistics of snippets as general framework for model-free entropy estimators, Phys. Rev. Lett. 130, 257101 (2023).
  22. J. Degünther, J. van der Meer, and U. Seifert, Fluctuating entropy production on the coarse-grained level: Inference and localization of irreversibility, Phys. Rev. Res. 6, 023175 (2024).
  23. K. Kanazawa and A. Dechant, Stochastic thermodynamics for classical non-Markov jump processes, arXiv:2506.04726.
  24. J. M. R. Parrondo, J. M. Horowitz, and T. Sagawa, Thermodynamics of information, Nat. Phys. 11, 131 (2015).
  25. T. Sagawa and M. Ueda, Generalized Jarzynski equality under nonequilibrium feedback control, Phys. Rev. Lett. 104, 090602 (2010).
  26. T. Sagawa and M. Ueda, Fluctuation theorem with information exchange: Role of correlations in stochastic thermodynamics, Phys. Rev. Lett. 109, 180602 (2012).
  27. T. Sagawa and M. Ueda, Nonequilibrium thermodynamics of feedback control, Phys. Rev. E 85, 021104 (2012).
  28. S. Still, D. A. Sivak, A. J. Bell, and G. E. Crooks, Thermodynamics of prediction, Phys. Rev. Lett. 109, 120604 (2012).
  29. S. Ito and T. Sagawa, Information thermodynamics on causal networks, Phys. Rev. Lett. 111, 180603 (2013).
  30. J. M. Horowitz and M. Esposito, Thermodynamics with continuous information flow, Phys. Rev. X 4, 031015 (2014).
  31. S. Ito, Backward transfer entropy: Informational measure for detecting hidden Markov models and its interpretations in thermodynamics, gambling and causality, Sci. Rep. 6, 36831 (2016).
  32. D. H. Wolpert and A. Kolchinsky, Thermodynamics of computing with circuits, New J. Phys. 22, 063047 (2020).
  33. G. Manzano, D. Subero, O. Maillet, R. Fazio, J. P. Pekola, and É. Roldán, Thermodynamics of gambling demons, Phys. Rev. Lett. 126, 080603 (2021).
  34. G. Manzano, G. Kardeş, É. Roldán, and D. H. Wolpert, Thermodynamics of computations with absolute irreversibility, unidirectional transitions, and stochastic computation times, Phys. Rev. X 14, 021026 (2024).
  35. D. H. Wolpert, J. Korbel, C. W. Lynn, F. Tasnim, J. A. Grochow, G. Kardeş, J. B. Aimone, V. Balasubramanian, E. De Giuli, D. Doty, N. Freitas, M. Marsili, T. E. Ouldridge, A. W. Richa, P. Riechers, É. Roldán, B. Rubenstein, Z. Toroczkai, and J. Paradiso, Is stochastic thermodynamics the key to understanding the energy costs of computation?, Proc. Natl. Acad. Sci. USA 121, e2321112121 (2024).
  36. S. Goldt and U. Seifert, Stochastic thermodynamics of learning, Phys. Rev. Lett. 118, 010601 (2017).
  37. S. Rooke, D. Krotov, V. Balasubramanian, and D. H. Wolpert, Stochastic thermodynamics of associative memory, arXiv:2601.01253.
  38. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, Attention is all you need, in Advances in Neural Information Processing Systems 30 (Curran Associates, Inc., Red Hook, NY, 2017), pp. 5998–6008.
  39. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, Language models are unsupervised multitask learners, OpenAI Technical Report, 2019.
  40. T. Brown et al., Language models are few-shot learners, in Advances in Neural Information Processing Systems (Curran Associates, Inc., Red Hook, NY, 2020), Vol. 33, pp. 1877–1901.
  41. H. Ramsauer et al., Hopfield networks is all you need, in The 9th International Conference on Learning Representations (ICLR, 2021).
  42. J. L. Elman, Finding structure in time, Cogn. Sci. 14, 179 (1990).
  43. S. Hochreiter and J. Schmidhuber, Long short-term memory, Neural Comput. 9, 1735 (1997).
  44. R. E. Kalman, A new approach to linear filtering and prediction problems, J. Basic Eng. 82, 35 (1960).
  45. B. D. O. Anderson and J. B. Moore, Optimal Filtering (Prentice-Hall, Englewood Cliffs, NJ, 1979).
  46. A. Lindquist and G. Picci, Linear Stochastic Systems: A Geometric Approach to Modeling, Estimation and Identification (Springer, Berlin, 2015).
  47. A. Gu, K. Goel, and C. Ré, Efficiently modeling long sequences with structured state spaces, in International Conference on Learning Representations (ICLR, 2022).
  48. J. T. H. Smith, A. Warrington, and S. W. Linderman, Simplified state space layers for sequence modeling, in International Conference on Learning Representations (ICLR, 2023).
  49. A. Gu and T. Dao, Mamba: Linear-time sequence modeling with selective state spaces, in Conference on Language Modeling (COLM) (2024).
  50. K. Li, A. K. Hopkins, D. Bau, F. Viégas, H. Pfister, and M. Wattenberg, Emergent world representations: Exploring a sequence model trained on a synthetic task, in The 11th International Conference on Learning Representations (ICLR, 2023).
  51. W. Gurnee and M. Tegmark, Language models represent space and time, in The 12th International Conference on Learning Representations (ICLR, 2024).
  52. D. P. Kingma and M. Welling, Auto-encoding variational Bayes, in International Conference on Learning Representations (ICLR, 2014).
  53. M. D. Hoffman and M. J. Johnson, ELBO surgery: Yet another way to carve up the variational evidence lower bound, in Proceedings of the Workshop in Advances in Approximate Bayesian Inference (NIPS, Barcelona, Spain, 2016).
  54. A. A. Alemi et al., Fixing a broken ELBO, in Proceedings of the 35th International Conference on Machine Learning (ICML) (PMLR, 2018), pp. 159–168.
  55. T. Matsumoto and T. Sagawa, Role of sufficient statistics in stochastic thermodynamics and its implication to sensory adaptation, Phys. Rev. E 97, 042103 (2018).
  56. R. Kawai, J. M. R. Parrondo, and C. Van den Broeck, Dissipation: The phase-space perspective, Phys. Rev. Lett. 98, 080602 (2007).
  57. V. Papadopoulos, J. Wenger, and C. Hongler, Arrows of time for large language models, in Proceedings of the 41st International Conference on Machine Learning (ICML) (PMLR, 2024), pp. 39509–39528.
  58. S. Yu et al., Reverse modeling in large language models, in Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Association for Computational Linguistics, Kerrville, TX, 2025), Vol. 2, pp. 306–320.
  59. C. Moslonka, H. Randrianarivo, A. Garnier, and E. Malherbe, Learned hallucination detection in black-box LLMs using token-level entropy production rate, in Advances in Information Retrieval (ECIR 2026), Lecture Notes in Computer Science (Springer, Cham, 2026), Vol. 16483, pp. 115–130.
  60. R. G. James, N. Barnett, and J. P. Crutchfield, Information flows? A critique of transfer entropies, Phys. Rev. Lett. 116, 238701 (2016).
  61. M. Miliani et al., ExpliCa: Evaluating explicit causal reasoning in large language models, in Findings of the Association for Computational Linguistics: ACL, 2025 (Association for Computational Linguistics, Kerrville, TX, 2025), pp. 17335–17355.
  62. S. K. Mitter and N. J. Newton, Information and entropy flow in the Kalman–Bucy filter, J. Stat. Phys. 118, 145 (2005).
  63. N. J. Newton, Dual Kalman–Bucy filters and interactive entropy production, SIAM J. Control Optim. 45, 998 (2006).
  64. N. J. Newton, Dual nonlinear filters and entropy production, SIAM J. Control Optim. 46, 1637 (2007).
  65. N. J. Newton, Interactive statistical mechanics and nonlinear filtering, J. Stat. Phys. 133, 711 (2008).
  66. J. M. Horowitz and H. Sandberg, Second-law-like inequalities with information and their interpretations, New J. Phys. 16, 125007 (2014).
  67. H. Sandberg, J.-C. Delvenne, N. J. Newton, and S. K. Mitter, Maximum work extraction and implementation costs for nonequilibrium Maxwell's demons, Phys. Rev. E 90, 042119 (2014).
  68. K. Kumasaki, K. Tojo, T. Sagawa, and K. Funo, Thermodynamic uncertainty relation for feedback cooling, Phys. Rev. E 113, 024134 (2026).
  69. C. Godrèche and J. M. Luck, Characterising the nonequilibrium stationary states of Ornstein–Uhlenbeck processes, J. Phys. A: Math. Theor. 52, 035002 (2019).
  70. G. T. Landi, T. Tomé, and M. J. de Oliveira, Entropy production in linear Langevin systems, J. Phys. A: Math. Theor. 46, 395001 (2013).
  71. M. Gilson, E. Tagliazucchi, and R. Cofré, Entropy production of multivariate Ornstein–Uhlenbeck processes correlates with consciousness levels in the human brain, Phys. Rev. E 107, 024121 (2023).
  72. T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed. (Wiley-Interscience, Hoboken, NJ, 2006).
  73. G. Weiss, Time-reversibility of linear stochastic processes, J. Appl. Probab. 12, 831 (1975).
  74. H. Tong and Z. Zhang, On time-reversibility of multivariate linear processes, Stat. Sin. 15, 495 (2005).
  75. T. T. Georgiou and A. Lindquist, On time-reversibility of linear stochastic models, in Proceedings of the 19th IFAC World Congress (IFAC, Cape Town, 2014), pp. 10403–10408.
  76. J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, Deep unsupervised learning using nonequilibrium thermodynamics, in Proceedings of the 32nd International Conference on Machine Learning (ICML) (PMLR, 2015), pp. 2256–2265.
  77. J. Ho, A. Jain, and P. Abbeel, Denoising diffusion probabilistic models, in Advances in Neural Information Processing Systems (NeurIPS 2020) (Curran Associates, Inc., Red Hook, NY, 2020), Vol. 33, pp. 6840–6851.
  78. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, Score-based generative modeling through stochastic differential equations, in Proceedings of the 9th International Conference on Learning Representations (ICLR, 2021).
  79. B. D. O. Anderson, Reverse-time diffusion equation models, Stoch. Process. Appl. 12, 313 (1982).
  80. A. Premkumar, Neural entropy, in NeurIPS 2025 (spotlight) (Curran Associates, Inc., Red Hook, NY, 2025).
  81. A. C. Barato and U. Seifert, Thermodynamic uncertainty relation for biomolecular processes, Phys. Rev. Lett. 114, 158101 (2015).
  82. J. M. Horowitz and T. R. Gingrich, Thermodynamic uncertainty relations constrain non-equilibrium fluctuations, Nat. Phys. 16, 15 (2020).
  83. N. Shiraishi, K. Funo, and K. Saito, Speed limit for classical stochastic processes, Phys. Rev. Lett. 121, 070601 (2018).
  84. K. Ikeda, T. Uda, D. Okanohara, and S. Ito, Speed-accuracy relations for diffusion models: Wisdom from nonequilibrium thermodynamics and optimal transport, Phys. Rev. X 15, 031031 (2025).
  85. C. R. Shalizi and J. P. Crutchfield, Computational mechanics: Pattern and prediction, structure and simplicity, J. Stat. Phys. 104, 817 (2001).
  86. A. B. Boyd, D. Mandal, and J. P. Crutchfield, Thermodynamics of modularity: Structural costs beyond the Landauer bound, Phys. Rev. X 8, 031036 (2018).
  87. A. B. Boyd, D. Mandal, and J. P. Crutchfield, Leveraging environmental correlations: The thermodynamics of requisite variety, J. Stat. Phys. 167, 1555 (2017).
  88. T. Sagawa, Data for Stochastic thermodynamics for autoregressive generative models: A non-Markovian perspective, GitHub, 2026, https://github.com/taksagawa/Paper2026.
  89. J. G. Kemeny and J. L. Snell, Finite Markov Chains (Van Nostrand, Princeton, NJ, 1960).
  90. T. Wolf et al., Transformers: State-of-the-art natural language processing, in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (Association for Computational Linguistics, Stroudsburg, PA, 2020), pp. 38–45.
  91. OpenAI, GitHub repository openai/gpt-2, README, https://github.com/openai/gpt-2, 2026.
  92. Hugging Face, https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt2/tokenization_gpt2.py, 2026.
  93. Hugging Face, https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils_tokenizers.py, 2026.
  94. A. Yang et al., Qwen3 Technical Report, arXiv:2505.09388.
  95. Hugging Face, https://huggingface.co/Qwen/Qwen3-4B-Base, 2026.

Outline

Information

PRX Intelligence 1, 023013– Published 5 November, 2026
  • Received 16 April 2026
  • Accepted 10 September 2026
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation