- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Hamiltonian of polymatrix zero-sum games

Toshihiro Ota1,2,* and Yuma Fujimoto1,3,†

  • *Contact author: ota_toshihiro@cyberagent.co.jp
  • †Contact author: fujimoto.yuma1991@gmail.com
PDF

Phys. Rev. Research 7, 043333 – Published 26 December, 2025

DOI: https://doi.org/10.1103/3gfz-lg1q

Abstract

The understanding of a dynamical system's properties can be significantly advanced by establishing it as a Hamiltonian system and then systematically exploring its inherent symmetries. By formulating agents’ strategies and cumulative payoffs as canonically conjugate variables, we identify the Hamiltonian function that generates the dynamics of polymatrix zero-sum games. We reveal the symmetries of our Hamiltonian and derive the associated conserved quantities, showing how the conservation of probability and the invariance of the Fenchel coupling are intrinsically encoded within the system. Furthermore, we propose the dissipation Follow-the-Regularized-Leader (DFTRL) dynamics by introducing a perturbation that dissipates the Fenchel coupling, proving convergence to the Nash equilibrium and linking DFTRL to last-iterate convergent algorithms. Our results highlight the potential of Hamiltonian dynamics in uncovering the structural properties of learning dynamics in games and pave the way for broader applications of Hamiltonian dynamics in game theory and machine learning.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (65)

  1. D. Fudenberg and D. K. Levine, The Theory of Learning in Games (MIT press, Cambridge, 1998), Vol. 2.
  2. J. F. Nash, Equilibrium points in n-person games, Proc. Natl. Acad. Sci. USA 36, 48 (1950).
  3. P. Mertikopoulos, C. Papadimitriou, and G. Piliouras, Cycles in adversarial regularized learning, in Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms (SIAM, Philadelphia, 2018), pp. 2703–2717.
  4. S. Shalev-Shwartz and Y. Singer, Convex repeated games and fenchel duality, in Proceedings of the 20th International Conference on Neural Information Processing Systems (MIT Press, Cambridge, MA, USA, 2006), pp. 1265–1272.
  5. J. Hofbauer, Evolutionary dynamics for bimatrix games: A Hamiltonian system? J. Math. Biol. 34, 675 (1996).
  6. J. P. Bailey and G. Piliouras, Multi-agent learning in network zero-sum games is a Hamiltonian system, in Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems (2019), pp. 233–241.
  7. C. Daskalakis and C. H. Papadimitriou, On a network generalization of the minmax theorem, in International Colloquium on Automata, Languages, and Programming (Springer, 2009), pp. 423–434.
  8. Y. Cai and C. Daskalakis, On minmax theorems for multiplayer games, in Proceedings of the Twenty-Second Annual ACM-SIAM Symposium on Discrete Algorithms (SIAM, Philadelphia, 2011), pp. 217–234.
  9. Y. Cai, O. Candogan, C. Daskalakis, and C. Papadimitriou, Zero-sum polymatrix games: A generalization of minmax, Math. Oper. Res. 41, 648 (2016).
  10. We make comments on the paper by Bailey and Piliouras in Appendix  pp3.
  11. Y. Sato, E. Akiyama, and J. D. Farmer, Chaos in learning a simple two-person game, Proc. Natl. Acad. Sci. USA 99, 4748 (2002).
  12. G. Ostrovski and S. van Strien, Piecewise linear Hamiltonian flows associated to zero-sum games: Transition combinatorics and questions on ergodicity, Regul. Chaot. Dyn. 16, 128 (2011).
  13. S. van Strien, Hamiltonian flows with random-walk behaviour originating from zero-sum games and fictitious play, Nonlinearity 24, 1715 (2011).
  14. D. Balduzzi, S. Racaniere, J. Martens, J. Foerster, K. Tuyls, and T. Graepel, The mechanics of n-player differentiable games, in International Conference on Machine Learning (PMLR, 2018), pp. 354–363.
  15. A. Letcher, D. Balduzzi, S. Racaniere, J. Martens, J. Foerster, K. Tuyls, and T. Graepel, Differentiable game mechanics, J. Mach. Learn. Res. 20, 3032 (2019).
  16. H. N. Alishah and P. Duarte, Hamiltonian evolutionary games, J. Dyn. Games 2, 33 (2015).
  17. A. H. Najafi, Conservative replicator and lotka-volterra equations in the context of Dirac∖big-isotropic structures, J. Geom. Mech. 12, 149 (2020).
  18. Y. Cai, A. Oikonomou, and W. Zheng, Finite-time last-iterate convergence for learning in multi-player games, in Advances in Neural Information Processing Systems (2022).
  19. C. Daskalakis and I. Panageas, Last-iterate convergence: Zero-sum games and constrained min-max optimization, in 10th Innovations in Theoretical Computer Science (2019).
  20. C. Daskalakis, A. Ilyas, V. Syrgkanis, and H. Zeng, Training gans with optimism, in International Conference on Learning Representations (2018).
  21. V. Syrgkanis, A. Agarwal, H. Luo, and R. E. Schapire, Fast convergence of regularized learning in games, in Proceedings of the 29th International Conference on Neural Information Processing Systems - Volume 2 (MIT Press, Cambridge, MA, USA, 2015), pp. 2989–2997.
  22. S. Rakhlin and K. Sridharan, Optimization, learning, and games with predictable sequences, in Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2 (Curran Associates Inc., Red Hook, NY, USA), pp. 3066–3074.
  23. S. Lee and D. Kim, Fast extra gradient methods for smooth structured nonconvex-nonconcave minimax problems, Adv. Neural Inf. Process. Syst. 34, 22588 (2021).
  24. P. Mertikopoulos, B. Lecouat, H. Zenati, C.-S. Foo, V. Chandrasekhar, and G. Piliouras, Optimistic mirror descent in saddle-point problems: Going the extra(-gradient) mile, in International Conference on Learning Representations (2019).
  25. G. M. Korpelevich, The extragradient method for finding saddle points and other problems, Ekon. mat. metod. 12, 747 (1976).
  26. R. A. Hemmat, A. Mitra, G. Lajoie, and I. Mitliagkas, LEAD: Min-max optimization from a physical perspective, ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems (2023).
  27. G. Zhang and Y. Wang, On the suboptimality of negative momentum for minimax optimization, in International Conference on Artificial Intelligence and Statistics (PMLR, 2021), pp. 2098–2106.
  28. N. B. Kovachki and A. M. Stuart, Continuous time analysis of momentum methods, J. Mach. Learn. Res. 22, 1 (2021).
  29. G. Gidel, R. A. Hemmat, M. Pezeshki, R. Le Priol, G. Huang, S. Lacoste-Julien, and I. Mitliagkas, Negative momentum for improved game dynamics, in The 22nd International Conference on Artificial Intelligence and Statistics (PMLR, 2019), pp. 1802–1811.
  30. B. T. Polyak, Some methods of speeding up the convergence of iteration methods, USSR Comput. Math. Math. Phys. 4, 1 (1964).
  31. Y. Feng, H. Fu, Q. Hu, P. Li, I. Panageas, B. Peng, and X. Wang, On the last-iterate convergence in time-varying zero-sum games: Extra gradient succeeds where optimism fails, Adv. Neural Inf. Process. Syst. 36, 21933 (2023).
  32. Y. Feng, P. Li, I. Panageas, and X. Wang, Last-iterate convergence separation between extra-gradient and optimism in constrained periodic games, Uncertainty in Artificial Intelligence (PMLR, 2024), pp. 1339–1370.
  33. The j=i term can be consistently included in the sum, since by our definition of Γ the corresponding payoff matrix is null; U(ii)≡0, which implies that agents do not engage in self-play (do not have self-interaction). The same applies in the subsequent sections generally, although we occasionally remove the j=i term from the sum for clarity.
  34. In this paper, we focus on games of Γσ. However, the discussion in this subsection is generically applicable to Γ, not only to Γσ.
  35. S. Shalev-Shwartz, Online learning and online convex optimization, Found. Trends Mach. Learn. 4, 107 (2012).
  36. P. D. Taylor and L. B. Jonker, Evolutionary stable strategies and game dynamics, Math. Biosci. 40, 145 (1978).
  37. P. D. Taylor, Evolutionarily stable strategies with two types of player, J. Appl. Probab. 16, 76 (1979).
  38. P. Schuster and K. Sigmund, Replicator dynamics, J. Theor. Biol. 100, 533 (1983).
  39. D. Friedman, Evolutionary games in economics, Econometrica 59, 637 (1991).
  40. A. Nagurney and D. Zhang, Projected dynamical systems in the formulation, stability analysis, and computation of fixed-demand traffic network equilibria, Transp. Sci. 31, 147 (1997).
  41. R. Lahkar and W. H. Sandholm, The projection dynamic and the geometry of population games, Games Econ. Behav. 64, 565 (2008).
  42. W. H. Sandholm, E. Dokumacı, and R. Lahkar, The projection dynamic and the replicator dynamic, Games Econ. Behav. 64, 666 (2008).
  43. P. Mertikopoulos and W. H. Sandholm, Learning in games via reinforcement and regularization, Math. Oper. Res. 41, 1297 (2016).
  44. P. Mertikopoulos and Z. Zhou, Learning in games with continuous action sets and unknown payoff functions, Math. Program. 173, 465 (2019).
  45. The code is available at https://github.com/Toshihiro-Ota/dftrl.
  46. J. Nash, Non-cooperative games, Ann. Math. 54, 286 (1951).
  47. E. Tewolde, B. H. Zhang, C. Oesterheld, T. Sandholm, and V. Conitzer, Computing game symmetries and equilibria that respect them, Proc. AAAI Conf. Artif. Intell. 39, 14148 (2025).
  48. J. Eisert, M. Wilkens, and M. Lewenstein, Quantum games and quantum strategies, Phys. Rev. Lett. 83, 3077 (1999).
  49. J. Eisert and M. Wilkens, Quantum games, J. Mod. Opt. 47, 2543 (2000).
  50. R. Jain, G. Piliouras, and R. Sim, Matrix multiplicative weights updates in quantum zero-sum games: Conservation laws & recurrence, Adv. Neural Inf. Process. Syst. 35, 4123 (2022).
  51. K. Lotidis, P. Mertikopoulos, and N. Bambos, Learning in quantum games, arXiv:2302.02333.
  52. W. Lin, G. Piliouras, R. Sim, and A. Varvitsiotis, Learning in quantum common-interest games and the separability problem, Quantum 9, 1689 (2025).
  53. R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud, Neural ordinary differential equations, Adv. Neural Inf. Process. Syst. (2018).
  54. R. T. Q. Chen, Torchdiffeq, GitHub (2018), https://github.com/rtqichen/torchdiffeq.
  55. D. P. Bertsekas, Nonlinear programming, J. Oper. Res. Soc. 48, 334 (1997).
  56. R. Bergmann and R. Herzog, Intrinsic formulation of KKT conditions and constraint qualifications on smooth manifolds, SIAM J. Optim. 29, 2423 (2019).
  57. P. Libermann and C.-M. Marle, Symplectic Geometry and Analytical Mechanics (Springer, Dordrecht, 1987).
  58. V. I. Arnol’d, Mathematical Methods of Classical Mechanics (Springer, New York, NY, 1989).
  59. In a narrow sense, the term “Hamiltonian dynamics” may be referred to as the solution to the flow equation  (B2) or (B3). However, we use this term collectively including the phase space, the Hamiltonian system, and its solutions, etc.
  60. In the literature, each time step of ε is implicitly assumed. Compared to our notation, our ϕ̂(nε) corresponds to their xn, etc.
  61. To be more precise, it should be α/T≪1, since the perturbative coefficient α is a dimensionful parameter.
  62. Since this paper has established the Hamiltonian structure for learning dynamics in games, also known as evolutionary dynamics, applying or extending our results to various analyses of Rock-Paper-Scissors games using physics-based approaches (such as Refs. [63, 64, 65]) could yield insights.
  63. K. M. A. Kabir and J. Tanimoto, The role of pairwise nonlinear evolutionary dynamics in the rock–paper–scissors game with noise, Appl. Math. Comput. 394, 125767 (2021).
  64. M. Tenorio, E. Rangel, and J. Menezes, Adaptive movement strategy in rock-paper-scissors models, Chaos Solitons Fractals 162, 112430 (2022).
  65. R. R. Justino, B. F. de Oliveira, and F. E. A. dos Santos, Critical phenomena in the rock-paper-scissors model, Physica A 680, 131018 (2025).

Outline

Information

Phys. Rev. Research 7, 043333– Published 26 December, 2025
  • Received 22 October 2025
  • Accepted 3 December 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation