- Physical Review Journals
All JournalsPhysics Magazine

PRX Intelligence

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Detailed Balance in Large-Language-Model-Driven Agents

Zhuo-Yang Song1,*, Qing-Hong Cao1,2,3,†, Ming-xing Luo4,‡, and Hua Xing Zhu1,3,§

  • *Contact author: zhuoyangsong@stu.pku.edu.cn
  • †Contact author: qinghongcao@pku.edu.cn
  • ‡Contact author: mingxingluo@csrc.ac.cn
  • §Contact author: zhuhx@pku.edu.cn
PDF

PRX Intelligence 1, 023008 – Published 27 October, 2026

DOI: https://doi.org/10.1103/kwx6-clqd

Abstract

Large language model (LLM)-driven agents are emerging as a powerful new paradigm for solving complex problems. Despite the empirical success of these practices, a theoretical framework to understand and unify their macroscopic dynamics remains lacking. This paper proposes a method based on the least action principle to estimate the underlying generative directionality of LLMs embedded within agents. By experimentally measuring the transition probabilities between LLM-generated states, we report evidence consistent with a tendency toward detailed balance in LLM-generated transitions, indicating that LLM generation may not be achieved by generally learning rule sets and strategies, but rather by implicitly learning a class of underlying potential functions that may transcend different LLM architectures and prompt templates. To our knowledge, this provides an effective macroscopic description of LLM generative dynamics that does not depend on specific model details. This work is an attempt to establish a macroscopic dynamics theory of complex AI systems, aiming to elevate the study of AI agents from a collection of engineering practices to a science built on effective measurements that are predictable and quantifiable.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (67)

  1. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou, Chain-of-thought prompting elicits reasoning in large language models, in Advances in Neural Information Processing Systems, edited by S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Curran Associates, Inc., Red Hook, NY, 2022), Vol. 35, pp. 24824–24837.
  2. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, ReAct: Synergizing reasoning and acting in language models (ICLR, 2023), https://openreview.net/forum?id=WE_vluYUL-X.
  3. S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan, Tree of Thoughts: Deliberate problem solving with large language models, Adv. Neural Inf. Process. Syst. 36, 11809 (2023).
  4. X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, Self-consistency improves chain of thought reasoning in language models (ICLR, 2023), https://openreview.net/forum?id=1PL1NIMMrw.
  5. L. Wang, W. Xu, Y. Lan, Z. Hu, Y. Lan, R. K.-W. Lee, and E.-P. Lim, Plan-and-solve prompting improving zero-shot chain-of-thought reasoning by large language models, in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, Vol. 1: Long Papers (2023), pp. 2609–2634.
  6. T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, Toolformer: Language models can teach themselves to use tools, Adv. Neural Inf. Process. Syst. 36, 68539 (2023).
  7. G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar, Voyager: An open-ended embodied agent with large language models, Trans. Mach. Learn. Res. (2024).
  8. W. Chen, X. Ma, X. Wang, and W. W. Cohen, Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks, Trans. Mach. Learn. Res. (2023).
  9. L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig, PAL program-aided language models, in Proceedings of the 40th International Conference on Machine Learning, edited by A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (PMLR, 2023), Vol. 202, pp. 10764–10799.
  10. J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, Code as policies language model programs for embodied control, in Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA) (IEEE, Piscataway, NJ, 2023), pp. 9493–9500.
  11. D. J. Mankowitz, A. Michi, A. Zhernov, M. Gelmi, M. Selvi, C. Paduraru, E. Leurent, S. Iqbal, J.-B. Lespiau, A. Ahern, et al., Faster sorting algorithms discovered using deep reinforcement learning, Nature (London) 618, 257 (2023).
  12. J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, Generative agents interactive simulacra of human behavior, in Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST’23 (ACM Press, New York, 2023).
  13. M. Besta, N. Blach, A. Kubicek, R. Gerstenberger, M. Podstawski, L. Gianinazzi, J. Gajda, T. Lehmann, H. Niewiadomski, P. Nyczyk, and T. Hoefler, Graph of thoughts: Solving elaborate problems with large language models, Proc. AAAI Conf. Artif. Intell. 38, 17682 (2024).
  14. M. Mohammadi, Y. Li, J. Lo, and W. Yip, Evaluation and benchmarking of LLM agents: A survey, in Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, KDD’25 (ACM Press, New York, 2025), pp. 6129–6139.
  15. R. Sapkota, K. I. Roumeliotis, and M. Karkee, AI agents vs. agentic AI: A conceptual taxonomy, applications and challenges, Inf. Fusion 126, 103599 (2026).
  16. D. A. Boiko, R. MacKnight, B. Kline, and G. Gomes, Autonomous chemical research with large language models, Nature (London) 624, 570 (2023).
  17. A. M. Bran, S. Cox, O. Schilter, C. Baldassari, A. D. White, and P. Schwaller, Augmenting large language models with chemistry tools, Nat. Mach. Intell. 6, 525 (2024).
  18. B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. R. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, P. Kohli, and A. Fawzi, Mathematical discoveries from program search with large language models, Nature (London) 625, 468 (2024).
  19. A. Novikov, N. Vũ, M. Eisenberger, E. Dupont, P.-S. Huang, A. Z. Wagner, S. Shirobokov, B. Kozlovskii, F. J. R. Ruiz, A. Mehrabian, M. P. Kumar, A. See, S. Chaudhuri, G. Holland, A. Davies, S. Nowozin, P. Kohli, and M. Balog, AlphaEvolve: A coding agent for scientific and algorithmic discovery, arXiv:2506.13131.
  20. Z.-Y. Song, Z. Cai, S. Zhang, J. Wei, J. Pan, Shi Qiu, Q.-H. Cao, T.-J. Hou, X. Liu, M.-x. Luo, and H. X. Zhu, Iterated agent for symbolic regression, Commun. Theor. Phys. (2026), doi:10.1088/1572-9494/aea105.
  21. N. Bhattacharya, N. Thomas, R. Rao, J. Dauparas, P. K. Koo, D. Baker, Y. S. Song, and S. Ovchinnikov, Interpreting Potts and transformer protein models through the lens of simplified attention, in Pacific Symposium on Biocomputing (World Scientific, Singapore, 2022), Vol. 27, pp. 34–45.
  22. Y. Sun and B. Haghighat, Phase transitions in large language models and the O(N) model, arXiv:2501.16241.
  23. Z. Liu, Y. Liu, J. Gore, and M. Tegmark, Neural thermodynamic laws for large language model training, arXiv:2505.10559.
  24. E. P. Hoel, When the map is better than the territory, Entropy 19, 188 (2017).
  25. J. J. Hopfield, Neural networks and physical systems with emergent collective computational abilities, Proc. Natl. Acad. Sci. USA 79, 2554 (1982).
  26. J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus, Emergent abilities of large language models, Trans. Mach. Learn. Res. (2022), https://openreview.net/forum?id=yzkSU5zdwD.
  27. R. Schaeffer, B. Miranda, and S. Koyejo, Are emergent abilities of large language models a mirage? in Advances in Neural Information Processing Systems, edited by A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Curran Associates, Inc., Red Hook, NY, 2023), Vol. 36, pp. 55565–55581.
  28. Y. Kim, K. Gu, C. Park, C. Park, S. Schmidgall, A. A. Heydari, Y. Yan, Z. Zhang, Y. Zhuang, Y. Liu, M. Malhotra, P. P. Liang, H. W. Park, Y. Yang, X. Xu, Y. Du, S. Patel, T. Althoff, D. McDuff, and X. Liu, Capable language models can outgrow the benefits of collaboration, Nat. Mach. Intell. 8, 1157 (2026).
  29. S. Hu, X. Cai, Y. Huang, Z. Yao, L. Zhang, P. Zhang, Y. Deng, and K. Chen, Emergent slow thinking in LLMs as inverse tree freezing, arXiv:2509.23629.
  30. A. Markov, Extension of the law of large numbers to quantities, depending on each other, J. Électron. Hist. Probab. Stat. 2, 10 (2006).
  31. N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller, Equation of state calculations by fast computing machines, J. Chem. Phys. 21, 1087 (1953).
  32. M. Mezard, G. Parisi, and M. Virasoro, Spin Glass Theory and Beyond: An Introduction to the Replica Method and Its Applications (World Scientific, Singapore, 1986).
  33. D. V. Schroeder, An Introduction to Thermal Physics (Oxford University Press, Oxford, 2021).
  34. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction (MIT Press, Cambridge, MA, 2018).
  35. N. Tishby and N. Zaslavsky, Deep learning and the information bottleneck principle, in 2015 IEEE Information Theory Workshop (ITW) (IEEE, Piscataway, NJ, 2015), pp. 1–5.
  36. N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang, On large-batch training for deep learning: Generalization gap and sharp minima, in International Conference on Learning Representations (OpenReview.net, 2017).
  37. H. Goldstein, C. P. Poole, and J. L. Safko, Classical Mechanics (Addison-Wesley, San Francisco, CA, 2002), 3rd ed.
  38. K. Friston, The free-energy principle: A unified brain theory? Nat. Rev. Neurosci. 11, 127 (2010).
  39. A. Tschantz, M. Baltieri, A. K. Seth, and C. L. Buckley, Scaling active inference, in 2020 International Joint Conference on Neural Networks (IJCNN) (IEEE, Piscataway, NJ, 2020), pp. 1–8.
  40. D. B. West, Introduction to Graph Theory (Prentice Hall, Upper Saddle River, NJ, 2018), 2nd ed.
  41. M. Yasunaga, X. Chen, Y. Li, P. Pasupat, J. Leskovec, P. Liang, E. H. Chi, and D. Zhou, Large language models as analogical reasoners, in Proceedings of the International Conference on Learning Representations (ICLR, 2024), Vol. 2024, pp. 17019–17045.
  42. See Supplemental Material at https://link.aps.org/supplemental/10.1103/kwx6-clqd for detailed derivations, agent implementation details, additional experiments, and supplementary figures, which includes Refs. [58, 59, 60, 61, 62, 63, 64, 65, 66, 67].
  43. T. Joachims, Optimizing search engines using clickthrough data, in Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD’02 (ACM Press, New York, 2002), pp. 133–142.
  44. X. Jiang, L.-H. Lim, Y. Yao, and Y. Ye, Statistical ranking and combinatorial Hodge theory, Math. Program. 127, 203 (2011).
  45. We show in Sec. S1 of the Supplemental Material [42] that the variational principle is equivalent to the least action principle under the condition that K(x) is a convex function.
  46. A. M. Lyapunov, The general problem of the stability of motion, Int. J. Control 55, 531 (1992).
  47. S. H. Strogatz, Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry, and Engineering (CRC Press, Boca Raton, FL, 2015), 2nd ed.
  48. E. Meyerson, G. Paolo, R. Dailey, H. Shahrzad, O. Francon, C. F. Hayes, X. Qiu, B. Hodjat, and R. Miikkulainen, Solving a million-step LLM task with zero errors, arXiv:2511.09030.
  49. With the Markov chain defined, time-reversal symmetry is well defined on its transition kernel as the equality of forward and reverse transition weights around every closed path, equivalent to detailed balance.
  50. S. Weinberg, Phenomenological Lagrangians, Physica A 96, 327 (1979).
  51. OpenAI, GPT-5 system card, Technical Report, OpenAI, 2025.
  52. Anthropic, System card: Claude Sonnet 4.5, Technical Report, Anthropic, 2025.
  53. G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al., Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, arXiv:2507.06261.
  54. J. C. Schön, Preferential trapping on energy landscapes in regions containing deep-lying minima: The reason for the success of simulated annealing? J. Phys. A: Math. Gen. 30, 2367 (1997).
  55. Ideasearch Collaboration, Ideasearch, 2025, https://github.com/IdeaSearch/IdeaSearch-framework.
  56. Z.-Y. Song, Q.-H. Cao, M.-x. Luo, and H. X. Zhu, Detailed balance in large language model-driven agents: Code, GitHub software repository, 2025, https://github.com/SonnyNondegeneracy/detialed-balance-llm, MIT license.
  57. Z.-Y. Song, Q.-H. Cao, M.-x. Luo, and H. X. Zhu, Detailed balance in large language model-driven agents: Data, Hugging Face dataset, 2026, https://huggingface.co/datasets/Nondegeneracy/detailed-balance-llm.
  58. X. Cai, S. Hu, T. Wang, Y. Huang, P. Zhang, Y. Deng, and K. Chen, Learning-at-criticality in large language models for quantum field theory and beyond, Chin. Phys. Lett. 42, 120002 (2025).
  59. Q.-H. Cao, Z.-Y. Hou, Y.-Y. Li, X. Liu, Z.-Y. Song, L.-Q. Zhang, S. Zhang, and K. Zhao, Scalable quantum state preparation via large-language-model-driven discovery, arXiv:2505.06347.
  60. K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, et al., Large language models encode clinical knowledge, Nature (London) 620, 172 (2023).
  61. Z.-Y. Song, T.-Z. Yang, Q.-H. Cao, M.-x. Luo, and H. X. Zhu, Explainable AI-assisted optimization for Feynman integral reduction, J. High Energy Phys. 06 (2026) 225.
  62. N. J. Szymanski, B. Rendy, Y. Fei, R. E. Kumar, T. He, D. Milsted, M. J. McDermott, M. Gallant, E. D. Cubuk, A. Merchant, H. Kim, A. Jain, C. J. Bartel, K. Persson, Y. Zeng, and G. Ceder, An autonomous laboratory for the accelerated synthesis of inorganic materials, Nature (London) 624, 86 (2023).
  63. Q. Chen, Y. Du, Z. Li, J. Liu, S. Duan, J. Guo, M. Liu, J. Liu, T. Yang, G. Zhang, L. Qin, W. Che, and W. Huang, The molecular structure of thought: Mapping the topology of long chain-of-thought reasoning, arXiv:2601.06002.
  64. A. Gladstone, G. Nanduru, M. M. Islam, P. Han, H. Ha, A. Chadha, Y. Du, H. Ji, J. Li, and T. Iqbal, Energy-based transformers are scalable learners and thinkers, arXiv:2507.02092.
  65. IdeaSearch Collaboration, IdeaSearchFitter, 2025, https://github.com/IdeaSearch/IdeaSearch-fit.
  66. R. S. Olson, W. La Cava, P. Orzechowski, R. J. Urbanowicz, and J. H. Moore, PMLB: A large benchmark suite for machine learning evaluation and comparison, BioData Mining 10, 36 (2017).
  67. Q. Team, Qwen2.5 technical report, arXiv:2412.15115.

Outline

Information

PRX Intelligence 1, 023008– Published 27 October, 2026
  • Received 3 January 2026
  • Accepted 15 September 2026
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation