- Open Access
Information maximization for generalized multiarmed bandit games
Phys. Rev. Research 8, 013047 – Published 20 January, 2026
DOI: https://doi.org/10.1103/wkw4-94hz
Abstract
Information and free-energy maximization are physics principles that provide general rules for an agent to optimize actions in line with specific goals and policies. These principles can serve as building blocks for designing decision-making policies capable of efficient performance with only partial information. Notably, the information-maximization principle has shown remarkable success in the classical bandit problem and has recently been shown to yield optimal algorithms for Gaussian and sub-Gaussian reward distributions. This article explores a broad extension of physics-based approaches to more complex and structured bandit problems. To this end, we cover three distinct types of bandit problems, where information maximization is adapted and leads to strong performance for Gaussian and Bernoulli reward distributions. Since the main challenge of information maximization lies in avoiding overexploration, we highlight how information is tailored at various levels to mitigate this issue, paving the way for more efficient and robust decision-making strategies.
Physics Subject Headings (PhySH)
Article Text
References (105)
- R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, Adaptive Computation and Machine Learning Series (A Bradford Book, Cambridge, MA, 1998).
- J. I. Gold and M. N. Shadlen, The neural basis of decision making, Annu. Rev. Neurosci. 30, 535 (2007).
- M. Jepma and S. Nieuwenhuis, Pupil diameter predicts changes in the exploration–exploitation trade-off: Evidence for the adaptive gain theory, J. Cognit. Neurosci. 23, 1587 (2011).
- H. Kim, J. H. Sul, N. Huh, D. Lee, and M. W. Jung, Role of striatum in updating values of chosen actions, J. Neurosci. 29, 14701 (2009).
- M. Vergassola, E. Villermaux, and B. I. Shraiman, ‘Infotaxis’ as a strategy for searching without gradients, Nature (London) 445, 406 (2007).
- J. Murlis, J. S. Elkinton, and R. T. Cardé, Odor plumes and how insects use them, Annu. Rev. Entomol. 37, 505 (1992).
- J.-B. Masson, M. B. Bechet, and M. Vergassola, Chasing information to search in random environments, J. Phys. A: Math. Theor. 42, 434009 (2009).
- Z. Wang and S. Jegelka, Max-value entropy search for efficient bayesian optimization, in Proceedings of the 34th International Conference on Machine Learning (Microtome Publishing, Paris, 2017), pp. 3627–3635.
- D. Russo and B. Van Roy, Learning to optimize via information-directed sampling, in Advances in Neural Information Processing Systems, edited by Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (Curran Associates, Inc., Red Hook, New York, 2014), Vol. 27, pp. 1583–1591.
- D. Lindner, M. Turchetta, S. Tschiatschek, K. Ciosek, and A. Krause, Information directed reward learning for reinforcement learning, 35th Conference on Neural Information Processing Systems (NeurIPS) (Curran Associates, Inc., Red Hook, New York, 2021).
- R. Houthooft, X. Chen, X. Chen, Y. Duan, J. Schulman, F. D. Turck, and P. Abbeel, VIME: Variational information maximizing exploration, Advances in Neural Information Processing Systems (NIPS), Vol. 29 (Curran Associates, Inc., Red Hook, New York, 2016), pp. 1109–1117.
- K. Ding, J. Li, and H. Liu, Interactive anomaly detection on attributed networks, in Proceedings of the 12th ACM International Conference on Web Search and Data Mining, WSDM ’19 (Association for Computing Machinery, New York, 2019), pp. 357–365.
- S. Zhang, D. Martinez, and J.-B. Masson, Multi-Robot searching with sparse binary cues and limited space perception, Front. Rob. AI 2, 317 (2015).
- D. Hernandez-Lobato, J. Hernandez-Lobato, A. Shah, and R. Adams, Predictive entropy search for multi-objective bayesian optimization, in Proceedings of the 33rd International Conference on Machine Learning, edited by M. F. Balcan and K. Q. Weinberger (PMLR, New York, 2016), Vol. 48, pp. 1492–1501.
- D. Martinez, L. Arhidi, E. Demondion, J. Masson, and P. Lucas, Using insect electroantennogram sensors on autonomous robots for olfactory searches, J. Vis. Exp. 90, e51704 (2014).
- R. T. Cardé, Navigation along windborne plumes of pheromone and resource-linked odors, Annu. Rev. Entomol. 66, 317 (2021).
- J. D. Cohen, S. M. McClure, and A. J. Yu, Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration, Philos. Trans. R. Soc. B 362, 933 (2007).
- K. Doya, Bayesian Brain: Probabilistic Approaches to Neural Coding (MIT Press, Cambridge, 2007).
- T. T. Hills, P. M. Todd, D. Lazer, A. D. Redish, and I. D. Couzin, Exploration versus exploitation in space, mind, and society, Trends Cognit. Sci. 19, 46 (2015).
- K. Friston, T. FitzGerald, F. Rigoli, P. Schwartenbeck, J. O Doherty, and G. Pezzulo, Active inference and learning, Neurosci. Biobehav. Rev. 68, 862 (2016).
- N. Tishby and N. Zaslavsky, Deep learning and the information bottleneck principle, IEEE Information Theory Workshop (ITW) (IEEE, Piscataway, New Jersey, 2015).
- M. Helias and D. Dahmen, Statistical Field Theory for Neural Networks, Lecture Notes in Physics (Springer International Publishing, Cham, 2020), Vol. 970.
- T. Parr, G. Pezzulo, and K. J. Friston, Active Inference: The Free Energy Principle in Mind, Brain, and Behavior (The MIT Press, Cambridge, 2022).
- J.-B. Masson, Olfactory searches with limited space perception, Proc. Natl. Acad. Sci. USA 110, 11261 (2013).
- G. Reddy, V. N. Murthy, and M. Vergassola, Olfactory sensing and navigation in turbulent environments, Annu. Rev. Condens. Matter Phys. 13, 191 (2022).
- T. Lattimore and C. Szepesvári, Bandit Algorithms (Cambridge University Press, Cambridge, 2020).
- D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, Mastering the game of go with deep neural networks and tree search, Nature (London) 529, 484 (2016).
- D. Bouneffouf, I. Rish, and C. Aggarwal, Survey on applications of multi-armed and contextual bandits, 2020 IEEE Congress on Evolutionary Computation (CEC) (IEEE, Piscataway, New Jersey, 2020), pp. 1–8.
- G. Burtini, J. Loeppky, and R. Lawrence, A survey of online experiment design with the stochastic multi-armed bandit, arXiv:1510.00757.
- A. Durand, C. Achilleos, D. Iacovides, K. Strati, G. D. Mitsis, and J. Pineau, Contextual bandits for adapting treatment in a mouse model of de novo carcinogenesis, in Proceedings of the 3rd Machine Learning for Healthcare Conference (Microtome Publishing, Paris, 2018), pp. 67–82.
- C. Gentile, S. Li, and G. Zappella, Online clustering of bandits, in Proceedings of the 31st International Conference on Machine Learning (Microtome Publishing, Paris, 2014), pp. 757–765.
- V. Kuleshov and D. Precup, Algorithms for multi-armed bandit problems, arXiv:1402.6028.
- J. Mary, R. Gaudel, and P. Preux, Bandits and recommender systems, in Machine Learning, Optimization, and Big Data, Lecture Notes in Computer Science, edited by P. Pardalos, M. Pavone, G. M. Farinella, and V. Cutello (Springer International Publishing, Cham, 2015), pp. 325–336.
- T. C. Stewart, T. Bekolay, and C. Eliasmith, Learning to select actions with spiking neurons in the basal ganglia, Front. Neurosci. 6, 1 (2012).
- T. L. Lai and H. Robbins, Asymptotically efficient adaptive allocation rules, Adv. Appl. Math. 6, 4 (1985).
- E. Kaufmann, N. Korda, and R. Munos, Thompson sampling: An asymptotically optimal finite-time analysis, in Algorithmic Learning Theory, Lecture Notes in Computer Science, edited by N. H. Bshouty, G. Stoltz, N. Vayatis, and T. Zeugmann (Springer, Berlin, 2012), pp. 199–213.
- P. Auer, Using upper confidence bounds for online learning, in Proceedings of the 41st Annual Symposium on Foundations of Computer Science (IEEE Computer Society, Redondo Beach, CA, 2000), pp. 270–279.
- S. Bubeck, R. Munos, and G. Stoltz, Pure exploration in finitely-armed and continuous-armed bandits, Theor. Comput. Sci. 412, 1832 (2011).
- K. Jamieson, M. Malloy, R. Nowak, and S. Bubeck, LIL’ UCB: An optimal exploration algorithm for multi-armed bandits, in Proceedings of the 27th Conference on Learning Theory (Microtome Publishing, Paris, 2014), pp. 423–439.
- E. Kaufmann and S. Kalyanakrishnan, Information complexity in bandit subset selection, in Proceedings of the 26th Annual Conference on Learning Theory (Microtome Publishing, Paris, 2013), pp. 228–251.
- S. Kalyanakrishnan, A. Tewari, P. Auer, and P. Stone, PAC subset selection in stochastic multi-armed bandits, in Proceedings of the 29th International Conference on Machine Learning, ICML 2012 (Microtome Publishing, Paris, 2012).
- R. Degenne and W. M. Koolen, Pure exploration with multiple correct answers, in Advances in Neural Information Processing Systems, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d’ Alché-Buc, E. Fox, and R. Garnett (Curran Associates, Inc., Red Hook, New York, 2019), Vol. 32, pp. 14591–14600.
- L. Li, W. Chu, J. Langford, and R. E. Schapire, A contextual-bandit approach to personalized news article recommendation, in Proceedings of the 19th International Conference on World Wide Web, WWW ’10 (Association for Computing Machinery, New York, 2010), pp. 661–670.
- B. Lin and D. Bouneffouf, Optimal epidemic control as a contextual combinatorial bandit with budget, 2022 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) (IEEE, Piscataway, New Jersey, 2022), pp. 1–8.
- L. Zhou, A survey on contextual multi-armed bandits, arXiv:1508.03326.
- P. Auer, Using confidence bounds for exploitation-exploration trade-offs, J. Mach. Learn. Res. 3, 397 (2002).
- T. Lattimore and C. Szepesvari, The end of optimism? An asymptotic analysis of finite-armed linear bandits, in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (Microtome Publishing, Paris, 2017), pp. 728–737.
- S. Agrawal and N. Goyal, Thompson sampling for contextual bandits with linear payoffs, in Proceedings of the 30th International Conference on Machine Learning (Microtome Publishing, Paris, 2013), pp. 127–135.
- S. S. Villar, J. Bowden, and J. Wason, Multi-armed bandit models for the optimal design of clinical trials: Benefits and challenges, Stat. Sci. 30, 199 (2015).
- S. S. Villar, Bandits strategies evaluated in the context of clinical trials in rare life-threatening diseases, Prob. Eng. Inf. Sci. 32, 229 (2018).
- R. de Heide, J. Cheshire, P. Ménard, and A. Carpentier, Bandits with many optimal arms, in Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21 (Curran Associates Inc., Red Hook, NY, 2024), pp. 22457–22469.
- T. Bonald and A. Proutiere, Two-target algorithms for infinite-armed bandits with bernoulli rewards, in Advances in Neural Information Processing Systems, edited by C. J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (Curran Associates, Inc., Red Hook, New York, 2013), Vol. 26, pp. 2184–2192.
- D. A. Berry, R. W. Chen, A. Zame, D. C. Heath, and L. A. Shepp, Bandit problems with infinitely many arms, Ann. Stat. 25, 2103 (1997).
- Y. Wang, J.-y. Audibert, and R. Munos, Algorithms for infinitely many-armed bandits, in Advances in Neural Information Processing Systems (Curran Associates, Inc., Red Hook, New York, 2008), Vol. 21, pp. 1729–1736.
- D. Baudry, E. Kaufmann, and O.-A. Maillard, Sub-sampling for efficient non-parametric bandit exploration, in Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20 (Curran Associates Inc., Red Hook, NY, 2020), pp. 5468–5478.
- S. Guha, K. Munagala, and P. Shi, Approximation algorithms for restless bandit problems, J. ACM 58, 1 (2010).
- N. Levine, K. Crammer, and S. Mannor, Rotting bandits, in Advances in Neural Information Processing Systems, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Curran Associates, Inc., Red Hook, NY, 2017), Vol. 30, pp. 3074–3083.
- P. Auer, N. Cesa-Bianchi, and P. Fischer, Finite-time analysis of the multiarmed bandit problem, Mach. Learn. 47, 235 (2002).
- H. Bastani, M. Bayati, and K. Khosravi, Mostly exploration-free algorithms for contextual bandits, Manage. Sci. 67, 1329 (2021).
- P. Ménard and A. Garivier, A minimax and asymptotically optimal algorithm for stochastic bandits, in Proceedings of the 28th International Conference on Algorithmic Learning Theory (Microtome Publishing, Paris, 2017), pp. 223–237.
- D. Russo and B. V. Roy, An information-theoretic analysis of thompson sampling, J. Mach. Learn. Res. 17, 1 (2016).
- G. Reddy, A. Celani, and M. Vergassola, Infomax strategies for an optimal balance between exploration and exploitation, J. Stat. Phys. 163, 1454 (2016).
- Y.-H. Hung, P.-C. Hsieh, X. Liu, and P. R. Kumar, Reward-biased maximum likelihood estimation for linear stochastic bandits, in Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35 (AAAI Press, 2021), pp. 7874–7882.
- A. Tirinzoni, M. Pirotta, M. Restelli, and A. Lazaric, An asymptotically optimal primal-dual incremental algorithm for contextual linear bandits, in Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20 (Curran Associates Inc., Red Hook, NY, 2020), pp. 1417–1427.
- A. Barbier-Chebbah, C. L. Vestergaard, and J.-B. Masson, Approximate information for efficient exploration-exploitation strategies, Phys. Rev. E 109, L052105 (2024).
- A. B. Barbier Chebbah, C. L. Vestergaard, J.-B. Masson, and E. Boursier, Approximate information maximization for bandit games, in Proceedings of the 28th International Conference on Artificial Intelligence and Statistics (Microtome Publishing, Paris, 2025), pp. 316–324.
- A. Slivkins, Introduction to multi-armed bandits, Found. Trends Mach. Learn. 12, 1 (2019).
- E. Kaufmann, O. Cappé, and A. Garivier, On the complexity of best-arm identification in multi-armed bandit models, J. Mach. Learn. Res. 17, 1 (2016).
- S. Kalyanakrishnan and P. Stone, Efficient selection of multiple bandit arms: Theory and practice, in Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML’10 (Omnipress, Madison, WI, 2010), pp. 511–518.
- S. Mannor and J. N. Tsitsiklis, The sample complexity of exploration in the multi-armed bandit problem, J. Mach. Learn. Res. 5, 623 (2004).
- Y. Zhou, X. Chen, and J. Li, Optimal PAC multiple arm identification with applications to crowdsourcing, in Proceedings of the 31st International Conference on Machine Learning (Microtome Publishing, Paris, 2014), pp. 217–225.
- S. Bubeck, T. Wang, and N. Viswanathan, Multiple identifications in multi-armed bandits, in Proceedings of the 30th International Conference on Machine Learning (Microtome Publishing, Paris, 2013), pp. 258–265.
- V. Gabillon, M. Ghavamzadeh, and A. Lazaric, Best arm identification: A unified approach to fixed budget and fixed confidence, in Advances in Neural Information Processing Systems edited by F. Pereira, C. J. Burges, L. Bottou, and K. Q. Weinberger (Curran Associates, Inc., Red Hook, New York, 2012), Vol. 25, pp. 3212–3220.
- J.-Y. Audibert and S. Bubeck, Best arm identification in multi-armed bandits, in COLT—23th Conference on Learning Theory (Omnipress, Madison, Wisconsin, 2010), p. 13.
- Note that is fixed and thus independent of the arms’ history.
- S. N. Majumdar, A. Pal, and G. Schehr, Extreme value statistics of correlated random variables: A pedagogical review, Phys. Rep. 840, 1 (2020).
- A. Carpentier and M. Valko, Simple regret for infinitely many armed bandits, in Proceedings of the 32nd International Conference on Machine Learning (Microtome Publishing, Paris, 2015), pp. 1133–1141.
- I. Rejwan and Y. Mansour, Top- combinatorial bandits with full-bandit feedback, in Proceedings of the 31st International Conference on Algorithmic Learning Theory (Microtome Publishing, Paris, 2020), pp. 752–776.
- S. Katariya, A. Tripathy, and R. Nowak, MaxGap bandit: Adaptive algorithms for approximate ranking, in Proceedings of the 33rd International Conference on Neural Information Processing Systems (Curran Associates, Inc., Red Hook, NY, 2019), pp. 11047–11057.
- A. Locatelli, M. Gutzeit, and A. Carpentier, An optimal algorithm for the thresholding bandit problem, in Proceedings of the 33rd International Conference on Machine Learning (Microtome Publishing, Paris, 2016), pp. 1690–1698.
- B. Mason, L. Jain, A. Tripathy, and R. Nowak, Finding all -good arms in stochastic bandits, in Advances in Neural Information Processing Systems edited by H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Curran Associates, Inc., Red Hook, NY, 2020), Vol. 33, pp. 20707–20718.
- W. Ren, J. Liu, and N. B. Shroff, Exploring out of top fraction of arms in stochastic bandits, in Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (Microtome Publishing, Paris, 2019), pp. 2820–2828.
- H. Zhuang, C. Wang, and Y. Wang, Identifying outlier arms in multi-armed bandit, in Advances in Neural Information Processing Systems, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Curran Associates, Inc., Red Hook, New York, 2017), Vol. 30, pp. 5204–5213.
- N. Abe, A. W. Biermann, and P. M. Long, Reinforcement learning with immediate rewards and linear hypotheses, Algorithmica 37, 263 (2003).
- W. Chu, L. Li, L. Reyzin, and R. Schapire, Contextual bandits with linear payoff functions, in Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (Microtome Publishing, Paris, 2011), pp. 208–214.
- A. Slivkins, Contextual bandits with similarity information, in Proceedings of the 24th Annual Conference on Learning Theory (Microtome Publishing, Paris, 2011), pp. 679–702.
- The definition of pseudoregret has not yet reached a consensus in the community. However, from a numerical perspective, all definitions agree on the objective of minimizing the average pseudoregret.
- R. Combes, S. Magureanu, and A. Proutiere, Minimal exploration in structured stochastic bandits, in Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17 (Curran Associates Inc., Red Hook, NY, 2017), pp. 1761–1769.
- V. Dani, T. P. Hayes, and S. M. Kakade, Stochastic linear optimization under bandit feedback, in 21st Annual Conference on Learning Theory—COLT 2008, Helsinki, Finland, 9–12 July 2008, edited by R. A. Servedio and T. Zhang (Omnipress, Madison, Wisconsin, 2008), pp. 355–366.
- Y. Abbasi-yadkori, D. Pál, and C. Szepesvári, Improved algorithms for linear stochastic bandits, in Advances in Neural Information Processing Systems (Curran Associates, Inc., Red Hook, New York, 2011), Vol. 24, pp. 2312–2320.
- B. Hao, T. Lattimore, and C. Szepesvari, Adaptive exploration in linear contextual bandit, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, Vol. 108 (PMLR, 2020), pp. 3536–3545.
- P. Rusmevichientong and J. N. Tsitsiklis, Linearly parameterized bandits, Math. Oper. Res. 35, 395 (2010).
- Note that these results rely on an adjustment of the variance of the posterior distribution.
- M. Abeille and A. Lazaric, Linear thompson sampling revisited, Electron. J. Stat. 11, 5165 (2017).
- J. Kirschner, T. Lattimore, C. Vernade, and C. Szepesvari, Asymptotically optimal information-directed sampling, in Proceedings of 34th Conference on Learning Theory (Microtome Publishing, Paris, 2021), pp. 2777–2821.
- J. Kirschner and A. Krause, Information directed sampling and bandits with heteroscedastic noise, in Proceedings of the 31st Conference on Learning Theory (Microtome Publishing, Paris, 2018), pp. 358–384.
- J. Kirschner, T. Lattimore, and A. Krause, Information directed sampling for linear partial monitoring, in Proceedings of 33rd Conference on Learning Theory (Microtome Publishing, Paris, 2020), pp. 2328–2369.
- M. Bayati, N. Hamidi, R. Johari, and K. Khosravi, Unreasonable effectiveness of greedy algorithms in multi-armed bandit with many arms, in Advances in Neural Information Processing Systems edited by H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Curran Associates, Inc., Red Hook, NY, 2020), Vol. 33, pp. 1713–1723.
- K. Jamieson and L. Jain, A bandit approach to multiple testing with false discovery control, in Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18 (Curran Associates Inc., Red Hook, NY, 2018), pp. 3664–3674.
- The lower bounds hold for sufficiently high values and 1-regular (), meaning that as goes to 0; see Ref. [98] for theorem details.
- Note that a sequential strategy has been proposed in Ref. [98], but this strategy still depends on a fixed bound on the subset size.
- https://gitlab.Pasteur.Fr/decision-making-related/papers/BroadBanditGame.
- E. S. Andersen, On the fluctuations of sums of random variables II, Math. Scand. 2, 195 (1954).
- M. Bauer, C. Godrèche, and J. M. Luck, Statistics of persistent events in the binomial random walk: Will the drunken sailor hit the sober man? J. Stat. Phys. 96, 963 (1999).
- P. Le Doussal and K. J. Wiese, Driven particle in a random landscape: Disorder correlator, avalanche distribution, and extreme value statistics of records, Phys. Rev. E 79, 051105 (2009).