- Physical Review Journals
All JournalsPhysics Magazine

PRX Life

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Optimality Theory of Stigmergic Collective Information Processing by Chemotactic Cells

Masaki Kato1,* and Tetsuya J. Kobayashi2,1,3,†

  • *Contact author: m_kato@sat.t.u-tokyo.ac.jp
  • †Contact author: tetsuya@mail.crmind.net
PDF

PRX Life 3, 043005 – Published 15 October, 2025

DOI: https://doi.org/10.1103/tvfy-lbbl

Abstract

Collective information processing is fundamental in various biological systems, where the cooperation of multiple cells results in complex functions beyond individual capabilities. A distinctive example is collective exploration in which chemotactic cells locate sparsely distributed targets by sharing information of them through active generation and modulation of endogenous chemical cues. While the optimality of sensing target-derived gradients has been studied extensively in the context of single-cell information processing, the optimality of collective information processing that self-organizes endogenous cue gradient towards the targets remains underexplored. In this study, we employ the reinforcement learning (RL) framework to formalize the collective exploration, in which a population adaptively optimize its navigation toward targets through active cue modulation. Based on RL theory, we derive the optimal exploration dynamics of agents and identify their structural correspondence with the Keller-Segel model, the established phenomenological model of collective cellular dynamics. Our theory identifies an optimal coupling relationship between self-organized gradient generation and sensing and demonstrates that the optimal cue-generation strategy changes qualitatively depending on whether cells sense gradients linearly or logarithmically. The underlying RL structure is leveraged to compare the derived collective dynamics with single-agent searching dynamics, showing that distributed information processing by the population enables a fraction of agents to reach the target robustly. Our formulation provides a foundation for understanding collective exploration mediated by active modulation and sensing of cues.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (100)

  1. J. A. Shapiro, Thinking about bacterial populations as multicellular organisms, Annu. Rev. Microbiol. 52, 81 (1998).
  2. R. Popat, D. M. Cornforth, L. McNally, and S. P. Brown, Collective sensing, and collective responses in quorum-sensing bacteria, J. R. Soc. Interface 12, 20140882 (2015).
  3. H.-C. Flemming, J. Wingender, U. Szewzyk, P. Steinberg, S. A. Rice, and S. Kjelleberg, Biofilms: An emergent form of bacterial life, Nat. Rev. Microbiol. 14, 563 (2016).
  4. P. Friedl and D. Gilmour, Collective cell migration in morphogenesis, regeneration, and cancer, Nat. Rev. Mol. Cell Biol. 10, 445 (2009).
  5. E. Theveneau and R. Mayor, Neural crest delamination, and migration: From epithelium-to-mesenchyme transition to collective cell migration, Dev. Biol. 366, 34 (2012).
  6. D. Ellison, A. Mugler, M. D. Brennan, S. H. Lee, R. J. Huebner, E. R. Shamir, L. A. Woo, J. Kim, P. Amar, I. Nemenman, A. J. Ewald, and A. Levchenko, Cell–cell communication enhances the capacity of cell ensembles to sense shallow gradients during morphogenesis, Proc. Natl. Acad. Sci. USA 113, E679 (2016).
  7. A. D. Luster, R. Alon, and U. H. von Andrian, Immune cell migration in inflammation: Present, and future therapeutic targets, Nat. Immunol. 6, 1182 (2005).
  8. A. Mantovani, M. A. Cassatella, C. Costantini, and S. Jaillon, Neutrophils in the activation, and regulation of innate, and adaptive immunity, Nat. Rev. Immunol. 11, 519 (2011).
  9. R. Axelrod, D. E. Axelrod, and K. J. Pienta, Evolution of cooperation among tumor cells, Proc. Natl. Acad. Sci. USA 103, 13474 (2006).
  10. T. S. Deisboeck and I. D. Couzin, Collective behavior in cancer cell populations, BioEssays 31, 190 (2009).
  11. R. H. Insall, P. Paschke, and L. Tweedy, Steering yourself by the bootstraps: How cells create their own gradients for chemotaxis, Trends Cell Biol. 32, 585 (2022).
  12. S. de Oliveira, E. E. Rosowski, and A. Huttenlocher, Neutrophil migration in infection, and wound repair: Going forward in reverse, Nat. Rev. Immunol. 16, 378 (2016).
  13. L. Tweedy, P. A. Thomason, P. I. Paschke, K. Martin, L. M. Machesky, M. Zagnoni, and R. H. Insall, Seeing around corners: Cells solve mazes, and respond at a distance using attractant breakdown, Science 369, eaay9792 (2020).
  14. B. Bréart, W. D. Ramos-Perez, A. Mendoza, A. K. Salous, M. Gobert, Y. Huang, R. H. Adams, J. J. Lafaille, D. Escalante-Alcalde, A. J. Morris, and S. R. Schwab, Lipid phosphate phosphatase 3 enables efficient thymic egress, J. Exp. Med. 208, 1267 (2011).
  15. T. Lämmermann, P. V. Afonso, B. R. Angermann, J. M. Wang, W. Kastenmüller, C. A. Parent, and R. N. Germain, Neutrophil swarms require LTB4, and integrins at sites of cell death in vivo, Nature (London) 498, 371 (2013).
  16. C. Scherber, A. J. Aranyosi, B. Kulemann, S. P. Thayer, M. Toner, O. Iliopoulos, and D. Irimia, Epithelial cell guidance by self-generated egf gradients, Integr. Biol. 4, 259 (2012).
  17. O. Susanto, Y. W. H. Koh, N. Morrice, S. Tumanov, P. A. Thomason, M. Nielson, L. Tweedy, A. J. Muinonen-Martin, J. J. Kamphorst, G. M. Mackay, and R. H. Insall, LPP3 mediates self-generation of chemotactic LPA gradients by melanoma cells, J. Cell Sci. 130, 3455 (2017).
  18. D. A. Clark and L. C. Grant, The bacterial chemotactic response reflects a compromise between transient, and steady-state behavior, Proc. Natl. Acad. Sci. USA 102, 9150 (2005).
  19. A. Celani and M. Vergassola, Bacterial strategies for chemotaxis response, Proc. Natl. Acad. Sci. USA 107, 1391 (2010).
  20. H. H. Mattingly, K. Kamino, B. B. Machta, and T. Emonet, Escherichia coli chemotaxis is information limited, Nat. Phys. 17, 1426 (2021).
  21. K. Nakamura and T. J. Kobayashi, Optimal sensing, and control of run-and-tumble chemotaxis, Phys. Rev. Res. 4, 013120 (2022).
  22. J. E. Segall, S. M. Block, and H. C. Berg, Temporal comparisons in bacterial chemotaxis, Proc. Natl. Acad. Sci. USA 83, 8987 (1986).
  23. Y. V. Kalinin, L. Jiang, Y. Tu, and M. Wu, Logarithmic sensing in Escherichia coli bacterial chemotaxis, Biophys. J. 96, 2439 (2009).
  24. M. D. Lazova, T. Ahmed, D. Bellomo, R. Stocker, and T. S. Shimizu, Response rescaling in bacterial chemotaxis, Proc. Natl. Acad. Sci. USA 108, 13870 (2011).
  25. Y. Tu, Quantitative modeling of bacterial chemotaxis: Signal amplification, and accurate adaptation, Annu. Rev. Biol. 42, 337 (2013).
  26. E. F. Keller and L. A. Segel, Model for chemotaxis, J. Theor. Biol. 30, 225 (1971).
  27. M. J. Tindall, P. K. Maini, S. L. Porter, and J. P. Armitage, Overview of mathematical approaches used to model bacterial chemotaxis II: Bacterial populations, Bull. Math. Biol. 70, 1570 (2008).
  28. K. J. Painter, Mathematical models for chemotaxis, and their applications in self-organisation phenomena, J. Theor. Biol. 481, 162 (2019).
  29. R. Alert, A. Martínez-Calvo, and S. S. Datta, Cellular sensing governs the stability of chemotactic fronts, Phys. Rev. Lett. 128, 148101 (2022).
  30. T. V. Phan, H. H. Mattingly, L. Vo, J. S. Marvin, L. L. Looger, and T. Emonet, Direct measurement of dynamic attractant gradients reveals breakdown of the Patlak–Keller–Segel chemotaxis model, Proc. Natl. Acad. Sci. USA 121, e2309251121 (2024).
  31. X. Fu, S. Kato, J. Long, H. H. Mattingly, C. He, D. C. Vural, S. W. Zucker, and T. Emonet, Spatial self-organization resolves conflicts between individuality, and collective migration, Nat. Commun. 9, 2177 (2018).
  32. H. H. Mattingly and T. Emonet, Collective behavior, and nongenetic inheritance allow bacterial populations to adapt to changing environments, Proc. Natl. Acad. Sci. USA 119, e2117377119 (2022).
  33. D. T. Fraebel, H. Mickalide, D. Schnitkey, J. Merritt, T. E. Kuhlman, and S. Kuehn, Environment determines evolutionary trajectory in a constrained phenotypic space, Elife 6, e24669 (2017).
  34. W. Liu, J. Cremer, D. Li, T. Hwa, and C. Liu, An evolutionarily stable strategy to colonize spatially extended habitats, Nature (London) 575, 664 (2019).
  35. J. Cremer, T. Honda, Y. Tang, J. Wong-Ng, M. Vergassola, and T. Hwa, Chemotaxis as a navigation strategy to boost range expansion, Nature (London) 575, 658 (2019).
  36. A. Pezzotta, M. Adorisio, and A. Celani, Chemotaxis emerges as the optimal solution to cooperative search games, Phys. Rev. E 98, 042401 (2018).
  37. D. Horstmann, and A. Stevens, A constructive approach to traveling waves in chemotaxis, J. Nonlinear Sci. 14, 1 (2004).
  38. T. E. J. Behrens, T. H. Muller, J. C. R. Whittington, S. Mark, A. B. Baram, K. L. Stachenfeld, and Z. Kurth-Nelson, What is a cognitive map? Organizing knowledge for flexible behavior, Neuron 100, 490 (2018).
  39. R. Solé, M. Moses, and S. Forrest, Liquid brains, solid brains, Philos. Trans. R. Soc. B 374, 20190040 (2019).
  40. M. G. Navas-Zuloaga, T. P. Pavlic, and B. H. Smith, Alternative model systems for cognitive variation: eusocial-insect colonies, Trends Cogn. Sci. 26, 836 (2022).
  41. E. Kolaczkowska and P. Kubes, Neutrophil recruitment, and function in health, and inflammation, Nat. Rev. Immunol. 13, 159 (2013).
  42. H. C. Berg, E. coli in Motion (Springer, Berlin, 2004).
  43. A. C. Szatmary, R. Nossal, C. A. Parent, and R. Majumdar, Modeling neutrophil migration in dynamic chemoattractant gradients: Assessing the role of exosomes during signal relay, Mol. Biol. Cell 28, 3457 (2017).
  44. S. Uderhardt, A. J. Martins, J. S. Tsang, T. Lämmermann, and R. N. Germain, Resident macrophages cloak tissue microlesions to prevent neutrophil-driven inflammatory damage, Cell 177, 541 (2019).
  45. L. G. Ng, J. S. Qin, B. Roediger, Y. Wang, R. Jain, L. L. Cavanagh, A. L. Smith, C. A. Jones, M. De Veer, M. A. Grimbaldeston, E. N. Meeusen, and W. Weninger, Visualizing the neutrophil response to sterile tissue injury in mouse dermis reveals a three-phase cascade of events, J. Invest. Dermatol. 131, 2058 (2011).
  46. P. V. Afonso, M. Janka-Junttila, Y. J. Lee, C. P. McCann, C. M. Oliver, K. A. Aamer, W. Losert, M. T. Cicerone, and C. A. Parent, LTB4 is a signal-relay molecule during neutrophil chemotaxis, Dev. Cell 22, 1079 (2012).
  47. K. M. Glaser, M. Mihlan, and T. Lämmermann, Positive feedback amplification in swarming immune cell populations, Curr. Opin. Cell Biol. 72, 156 (2021).
  48. S. X. Tamás, B. T. Roux, B. Vámosi, F. G. Dehne, A. Török, L. Fazekas, and B. Enyedi, A genetically encoded sensor for visualizing leukotriene b4 gradients in vivo, Nat. Commun. 14, 4610 (2023).
  49. T. Kato and T. J. Kobayashi, Understanding adaptive immune system as reinforcement learning, Phys. Rev. Res. 3, 013222 (2021).
  50. E. Todorov, Efficient computation of optimal actions, Proc. Natl. Acad. Sci. USA 106, 11478 (2009).
  51. See Supplemental Material at http://link.aps.org/supplemental/10.1103/tvfy-lbbl for additional details on RL theory, derivation, and experiments, which also contains Refs. [84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100].
  52. C. Janetopoulos, L. Ma, P. N. Devreotes, and P. A. Iglesias, Chemoattractant-induced phosphatidylinositol 3, 4, 5-trisphosphate accumulation is spatially amplified, and adapts, independent of the actin cytoskeleton, Proc. Natl. Acad. Sci. USA 101, 8951 (2004).
  53. Y. Tu and W.-J. Rappel, Adaptation in living systems, Annu. Rev. Condens. Matter Phys. 9, 183 (2018).
  54. V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, Asynchronous methods for deep reinforcementlearning, in Proceedings of The 33rd International Conference on Machine Learning (PMLR, Cambridge, MA, 2016), Vol. 48, pp. 1928–1937.
  55. T. J. Newman and R. Grima, Many-body theory of chemotactic cell-cell interactions, Phys. Rev. E 70, 051916 (2004).
  56. E. Strickland, D. Pan, C. Godfrey, J. S. Kim, A. Hopke, W. Ji, M. Degrange, B. Villavicencio, M. K. Mansour, C. S. Zerbe, D. Irimia, A. Amir, and O. D. Weiner, Self-extinguishing relay waves enable homeostatic control of human neutrophil swarming, Dev. Cell 59, 2659 (2024).
  57. D. Fuller, W. Chen, M. Adler, A. Groisman, H. Levine, W.-J. Rappel, and W. F. Loomis, External, and internal constraints on eukaryotic chemotaxis, Proc. Natl. Acad. Sci. USA 107, 9656 (2010).
  58. A. S. Perelson, Theoretical Immunology, Part Two (Addison-Wesley, Boston, 1988).
  59. R. Fox, A. Pakman, and N. Tishby, Taming the noise in reinforcement learning via soft updates, in Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence (AUAI Press, Arlington, VA, 2016).
  60. J. Monod, On the nature of allosteric transitions: A plausible model, J. Mol. Biol. 12, 88 (1965).
  61. A. D. Renault, P. S. Kunwar, and R. Lehmann, Lipid phosphate phosphatase activity regulates dispersal, and bilateral sorting of embryonic germ cells in Drosophila, Development 137, 1815 (2010).
  62. T. V. Phan, R. Morris, M. E. Black, T. K. Do, K.-C. Lin, K. Nagy, J. C. Sturm, J. Bos, and R. H. Austin, Bacterial route finding, and collective escape in mazes, and fractals, Phys. Rev. X 10, 031017 (2020).
  63. M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, 1st ed. (John Wiley & Sons, New York, 1994).
  64. W. F. Vining, F. Esponda, M. E. Moses, and S. Forrest, How does mobility help distributed systems compute? Philos. Trans. R. Soc. B 374, 20180375 (2019).
  65. M. E. Moses, J. L. Cannon, D. M. Gordon, and S. Forrest, Distributed adaptive search in T cells: Lessons from ants, Front. Immunol. 10, 1357 (2019).
  66. D. H. Wolpert and W. G. Macready, No Free Lunch Theorems for Search, Technical Report 12, Santa Fe Institute (1995).
  67. Y. Chow, O. Nachum, and M. Ghavamzadeh, Path consistency learning in Tsallis entropy regularized MDPs, in Proceedings of the 35th International Conference on Machine Learning (PMLR, Cambridge, MA, 2018), Vol. 80, pp. 979–988.
  68. K. Lang and G. E. Hinton, Dimensionality reduction, and prior knowledge in E-Set Recognition, in Advances in Neural Information Processing Systems (Morgan Kaufmann, San Francisco, CA, 1989), Vol. 2, pp. 178–185.
  69. X. Zhu, Z. Ghahramani, and J. D. Lafferty, Semi-supervised learning using Gaussian fields, and harmonic functions, in Proceedings of the 20th International Conference on Machine Learning (AAAI Press, Washington, DC, 2003), pp. 912–919.
  70. D. Chowdhury, A. Schadschneider, and K. Nishinari, Physics of transport, and traffic phenomena in biology: From molecular motors, and cells to organisms, Phys. Life Rev. 2, 318 (2005).
  71. J. Weickert, A review of nonlinear diffusion filtering, in Proceedings of International Conference on Scale-space Theories in Computer Vision (Springer, Berlin, 1997), pp. 1–28.
  72. C. E. Hughes and R. J. B. Nibbs, A guide to chemokines, and their receptors, FEBS J. 285, 2944 (2018).
  73. A. Hopke, A. Scherer, S. Kreuzburg, M. S. Abers, C. S. Zerbe, M. C. Dinauer, M. K. Mansour, and D. Irimia, Neutrophil swarming delays the growth of clusters of pathogenic fungi, Nat. Commun. 11, 2031 (2020).
  74. B. J. Crespi, The evolution of social behavior in microorganisms, Trends Ecol. Evol. 16, 178 (2001).
  75. B. Stramer and R. Mayor, Mechanisms and in vivo functions of contact inhibition of locomotion, Nat. Rev. Mol. Cell Biol. 18, 43 (2017).
  76. E. Theveneau, B. Steventon, E. Scarpa, S. Garcia, X. Trepat, A. Streit, and R. Mayor, Chase-and-run between adjacent cell populations promotes directional collective migration, Nat. Cell Biol. 15, 763 (2013).
  77. M. Laurière, S. Perrin, M. Geist, and O. Pietquin, Learning mean fieldgames: A survey, arXiv:2205.12944.
  78. S. G. Subramanian, P. Poupart, M. E. Taylor, and N. Hegde, Multi type mean field reinforcement learning, in Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '20 (IFAAMAS, Richland, SC, Auckland, New Zealand, 2020), pp. 411–419.
  79. Z. Schuss, K. Basnayake, and D. Holcman, Redundancy principle, and the role of extreme statistics in molecular, and cellular biology, Phys. Life Rev. 28, 52 (2019).
  80. P. Turney, D. Whitley, and R. W. Anderson, Evolution, learning, and instinct: 100 years of the Baldwin effect, Evol. Comput. 4, 271 (1996).
  81. B. Xue and S. Leibler, Evolutionary learning of adaptation to varying environments through a transgenerational feedback, Proc. Natl. Acad. Sci. USA 113, 11266 (2016).
  82. S. Khadka and K. Tumer, Evolution-guided policy gradient in reinforcement learning, in Advances in Neural Information Processing Systems (Curran Associates Inc., Red Hook, NY, 2018).
  83. S. Nakashima, and T. J. Kobayashi, Acceleration of evolutionary processes by learning, and extended Fisher's fundamental theorem, Phys. Rev. Res. 4, 013069 (2022).
  84. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction (MIT Press, 2018).
  85. S. C. Tanaka, K. Doya, G. Okada, K. Ueda, Y. Okamoto, and S. Yamawaki, Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops, Nat. Neurosci. 7, 887 (2004).
  86. J. E. M. Bennett, A. Philippides, and T. Nowotny, Learning with reinforcement prediction errors in a model of the Drosophila mushroom body, Nat. Commun. 12, 2569 (2021).
  87. W. Schultz, Dopamine reward prediction-error signalling: a two-component response, Nat. Rev. Neurosci. 17, 183 (2016).
  88. B. D. Ziebart, Modeling purposeful adaptive behavior with the principle of maximum causal entropy, Ph.D. thesis, Carnegie Mellon University, 2010.
  89. T. Haarnoja, H. Tang, P. Abbeel, and S. Levine, Reinforcement learning with deep energy-based policies, in Proceedings of the 34th International Conference on Machine Learning (PMLR, Cambridge, MA, 2017), Vol. 70, pp. 1352–1361.
  90. M. G. Azar, V. Gómez, and H. J. Kappen, Dynamic policy programming, J. Mach. Learn. Res. 13, 3207 (2012).
  91. W.-W. Tso and J. Adler, Negative chemotaxis in Escherichia coli, J. Bacteriol. 118, 560 (1974).
  92. D. Precup, R. S. Sutton, and S. P. Singh, Eligibility traces for off-policy policy evaluation, in Proceedings of the Seventeenth International Conference on Machine Learning (Morgan Kaufmann Publishers, San Francisco, CA, 2000), pp. 759–766.
  93. T. Sun, Y. Sun, and W. Yin, On Markov chain gradient descent, in Advances in Neural Information Processing Systems (Curran Associates, Red Hook, NY, 2018), pp. 9918–9927.
  94. V. S. Borkar, Stochastic Approximation: A Dynamical Systems Viewpoint (Springer, 2008).
  95. A. Stevens, The derivation of chemotaxis equations as limit dynamics of moderately interacting stochastic many-particle systems, SIAM J. Appl. Math. 61, 183 (2000).
  96. H. J. Kushner and P. Dupuis, Numerical Methods for Stochastic Control Problems in Continuous Time, 2nd ed. (Springer, 2001), Vol. 24.
  97. B. Oksendal, Stochastic Differential Equations: An Introduction with Applications, 3rd ed. (Springer-Verlag, 1992).
  98. R. Grima and T. J. Newman, Accurate discretization of advection-diffusion equations, Phys. Rev. E 70, 036703 (2004).
  99. H. Meyer and H. Rieger, Alignment interaction and band formation in assemblies of autochemorepulsive walkers, Phys. Rev. E 108, 034604 (2023).
  100. M. Kato and T. J. Kobayashi, Optimality theory of stigmergic collective information processing by chemotactic cells, arXiv:2407.15298.

Outline

Information

PRX Life 3, 043005– Published 15 October, 2025
  • Received 19 February 2025
  • Accepted 25 August 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation