- Physical Review Journals
All JournalsPhysics Magazine

PRX Life

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Optimal Control of Stochastic Reaction Networks with Entropic Control Cost and Emergence of Mode-Switching Strategies

Shuhei A. Horiguchi*,† and Tetsuya J. Kobayashi‡,§

  • Department of Mathematical Informatics, Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan

  • *Present address: Nano Life Science Institute, Kanazawa University, Kanazawa 920-1192, Japan.
  • †Contact author: shuhei.a.horiguchi@gmail.com
  • ‡Present address: Institute of Industrial Science, The University of Tokyo, Tokyo 153-8505, Japan and Universal Biology Institute, The University of Tokyo, Tokyo 113-8654, Japan.
  • §Contact author: tetsuya@mail.crmind.net
PDF

PRX Life 3, 033027 – Published 26 September, 2025

DOI: https://doi.org/10.1103/zttn-tpzq

Abstract

Controlling the stochastic dynamics of biological populations is a challenge that arises across various biological contexts. However, these dynamics are inherently nonlinear and involve a discrete state space, i.e., the number of molecules, cells, or organisms. Additionally, the possibility of extinction has a significant impact on both dynamics and control strategies, particularly when the population size is small. These factors hamper the direct application of conventional control theories to biological systems. To address these challenges, we formulate the optimal control problem for stochastic population dynamics by utilizing control cost functions based on the f-divergence, which naturally accounts for population-specific factors. If Kullback-Leibler divergence is adopted for the cost function, the complex nonlinear Hamilton-Jacobi-Bellman equation is simplified into a linear form, facilitating efficient computation of optimal solutions. We demonstrate the effectiveness of our approach by applying it to the control of interacting random walkers, Moran processes, and SIR models, and observe the mode-switching phenomena in the control strategies. Our approach provides new opportunities for applying control theory to a wide range of biological problems.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (87)

  1. P. C. Bressloff, Stochastic Processes in Cell Biology, Interdisciplinary Applied Mathematics, Vol. 41 (Springer, Cham, 2014).
  2. H. Qian and H. Ge, Stochastic Chemical Reaction Systems in Biology (Springer, Cham, 2021).
  3. J.-B. Lugagne, S. Sosa Carrillo, M. Kirch, A. Köhler, G. Batt, and P. Hersen, Balancing a genetic toggle switch by real-time feedback control and periodic forcing, Nat. Commun. 8, 1671 (2017).
  4. R. A. Gatenby, A. S. Silva, R. J. Gillies, and B. R. Frieden, Adaptive therapy, Cancer Res. 69, 4894 (2009).
  5. M. Sehl, H. Zhou, J. S. Sinsheimer, and K. L. Lange, Extinction models for cancer stem cell therapy, Math. Biosci. 234, 132 (2011).
  6. M. Raatz and A. Traulsen, Promoting extinction or minimizing growth? The impact of treatment on trait trajectories in evolving populations, Evolution 77, 1408 (2023).
  7. C. McKee and G. R. Chaudhry, Advances and challenges in stem cell culture, Colloids Surf. B 159, 62 (2017).
  8. A. C. Ericsson and C. L. Franklin, Manipulating the gut microbiota: Methods and challenges, ILAR J. 56, 205 (2015).
  9. D. L. Woodland and J. E. Kohlmeier, Migration, maintenance and recall of memory T cells in peripheral tissues, Nat. Rev. Immunol. 9, 153 (2009).
  10. F. Sallusto, A. Lanzavecchia, K. Araki, and R. Ahmed, From vaccines to memory and back, Immunity 33, 451 (2010).
  11. M. R. W. Rands, W. M. Adams, L. Bennun, S. H. M. Butchart, A. Clements, D. Coomes, A. Entwistle, I. Hodge, V. Kapos, J. P. W. Scharlemann, W. J. Sutherland, and B. Vira, Biodiversity conservation: Challenges beyond 2010, Science 329, 1298 (2010).
  12. S. Sarkar, R. L. Pressey, D. P. Faith, C. R. Margules, T. Fuller, D. M. Stoms, A. Moffett, K. A. Wilson, K. J. Williams, P. H. Williams, and S. Andelman, Biodiversity conservation planning tools: Present status and challenges for the future, Annu. Rev. Environ. Resour. 31, 123 (2006).
  13. M. Kuussaari, R. Bommarco, R. K. Heikkinen, A. Helm, J. Krauss, R. Lindborg, E. Öckinger, M. Pärtel, J. Pino, F. Rodà, C. Stefanescu, T. Teder, M. Zobel, and I. Steffan-Dewenter, Extinction debt: A challenge for biodiversity conservation, Trends Ecol. Evol. 24, 564 (2009).
  14. A. Fischer, I. Vázquez-García, and V. Mustonen, The value of monitoring to control evolving populations, Proc. Natl. Acad. Sci. USA 112, 1007 (2015).
  15. A. Nourmohammad and C. Eksin, Optimal evolutionary control for artificial selection on molecular phenotypes, Phys. Rev. X 11, 011044 (2021).
  16. J. Piret and G. Boivin, Pandemics throughout history, Front. Microbiol. 11, 631736 (2021).
  17. A. Aurell, R. Carmona, G. Dayanikli, and M. Laurière, Optimal incentives to mitigate epidemics: A Stackelberg mean field game approach, SIAM J. Control Optim. 60, S294 (2022).
  18. S. K. Schnyder, J. J. Molina, R. Yamamoto, and M. S. Turner, Rational social distancing policy during epidemics with limited healthcare capacity, PLoS Comput. Biol. 19, e1011533 (2023).
  19. F. Avram, R. Adenane, and M. Neagu, Advancing mathematical epidemiology and chemical reaction network theory via synergies between them, Entropy 26, 936 (2024).
  20. C. Gardiner, Stochastic Methods, 4th ed. (Springer, Berlin, 2009).
  21. D. F. Anderson and T. G. Kurtz, Continuous time Markov chain models for chemical reaction networks, in Design and Analysis of Biomolecular Circuits: Engineering Approaches to Systems and Synthetic Biology, edited by H. Koeppl, G. Setti, M. di Bernardo, and D. Densmore (Springer, New York, 2011), pp. 3–42.
  22. I. Csiszár, Information-type measures of difference of probability distributions and indirect, Stud. Sci. Math. Hung. 2, 299 (1967).
  23. T. Morimoto, Markov processes and the H-theorem, J. Phys. Soc. Jpn. 18, 328 (1963).
  24. Y. Polyanskiy and Y. Wu, Information Theory: From Coding to Learning (Cambridge University Press, Cambridge, 2025).
  25. W. H. Fleming and S. K. Mitter, Optimal control and nonlinear filtering for nondegenerate diffusion processes, Stochastics 8, 63 (1982).
  26. S. K. Mitter and N. J. Newton, A variational approach to nonlinear estimation, SIAM J. Control Optim. 42, 1813 (2003).
  27. E. Todorov, General duality between optimal control and estimation, in 2008 47th IEEE Conference on Decision and Control (IEEE, 2008), pp. 4286–4292.
  28. H. J. Kappen, Path integrals and symmetry breaking for optimal control theory, J. Stat. Mech. (2005) P11011.
  29. W. H. Fleming and H. M. Soner, Controlled Markov Processes and Viscosity Solutions, Stochastic Modelling and Applied Probability, Vol. 25 (Springer-Verlag, New York, 2006).
  30. E. Todorov, Efficient computation of optimal actions, Proc. Natl. Acad. Sci. USA 106, 11478 (2009).
  31. S. P. Boyd and L. Vandenberghe, Convex Optimization (Cambridge University Press, Cambridge, 2004).
  32. L. Lorch, A. De, S. Bhatt, W. Trouleau, U. Upadhyay, and M. Gomez-Rodriguez, Stochastic optimal control of epidemic processes in networks, arXiv:1810.13043.
  33. E. A. Theodorou and E. Todorov, Stochastic optimal control for nonlinear Markov jump diffusion processes, in 2012 American Control Conference (ACC) (IEEE, Montreal, QC, Canada, 2012), pp. 1633–1639.
  34. O. Yuta, K. Kashima, and Y. Ohta, Iterative path integral approach to nonlinear stochastic optimal control under compound poisson noise, Asian J. Control 19, 781 (2017).
  35. C. Briat and M. Khammash, Continuous-time and sampled-data optimal control of linear stochastic reaction networks, in 2022 IEEE 61st Conference on Decision and Control (CDC) (IEEE, Cancun, Mexico, 2022), pp. 3865–3870.
  36. C. Briat and M. Khammash, Continuous and sampled-data H∞ control of linear stochastic reaction networks, in 2022 IEEE 61st Conference on Decision and Control (CDC) (IEEE, Cancun, Mexico, 2022), pp. 1924–1929.
  37. C. Briat and M. Khammash, Optimal and H∞ control of stochastic reaction networks, arXiv:2111.14754.
  38. S. A. Horiguchi and T. J. Kobayashi, Cellular gradient flow structure linking single-cell-level rules and population-level dynamics, Phys. Rev. Res. 5, L022052 (2023).
  39. C. Kipnis and C. Landim, Scaling Limits of Interacting Particle Systems, Grundlehren der mathematischen Wissenschaften, Vol. 320 (Springer, Berlin, 1999).
  40. E. A. Theodorou and E. Todorov, Relative entropy and free energy dualities: Connections to path integral and KL control, in 2012 IEEE 51st IEEE Conference on Decision and Control (CDC) (IEEE, Maui, HI, USA, 2012), pp. 1466–1473.
  41. Y. Gao, J.-G. Liu, and O. Tse, Optimal control formulation of transition path problems for Markov jump processes, arXiv:2311.07795.
  42. O. Guéant, Optimal control on finite graphs: Asymptotic optimal controls and ergodic constant in the case of entropic costs, arXiv:2011.02212.
  43. S. Jaimungal, S. M. Pesenti, and L. Sánchez-Betancourt, Minimal Kullback–Leibler divergence for constrained Lévy–Itô processes, SIAM J. Control Optim. 62, 982 (2024).
  44. P. Xie, Theoretical analysis of dynamics of kinesin molecular motors, ACS Omega 5, 5721 (2020).
  45. W. Feller, Infinitely divisible distributions and bessel functions associated with random walks, SIAM J. Appl. Math. 14, 864 (1966).
  46. S. Karlin and J. McGregor, Coincidence probabilities, Pacific J. Math. 9, 1141 (1959).
  47. M. E. Fisher, Walks, walls, wetting, and melting, J. Stat. Phys. 34, 667 (1984).
  48. M. Katori and H. Tanemura, Noncolliding Brownian motion and determinantal processes, J. Stat. Phys. 129, 1233 (2007).
  49. P. A. P. Moran, The rate of approach to homozygosity, Ann. Human Genet. 23, 1 (1958).
  50. S. Karlin and J. McGregor, The number of mutants maintained in a population, in Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, edited by L. M. Le Cam and J. Neyman (University of California Press, Berkeley, 1967), Vol. 4, pp. 415–438.
  51. B. Houchmandzadeh and M. Vallade, Alternative to the diffusion equation in population genetics, Phys. Rev. E 82, 051913 (2010).
  52. P. Ashcroft, A. Traulsen, and T. Galla, When the mean is not enough: Calculating fixation time distributions in birth-death processes, Phys. Rev. E 92, 042154 (2015).
  53. D. Hershkowitz, V. Mehrmann, and H. Schneider, Eigenvalue interlacing for certain classes of matrices with real principal minors, Linear Algebra Appl. 88-89, 373 (1987).
  54. N. T. J. Bailey, The total size of a general stochastic epidemic, Biometrika 40, 177 (1953).
  55. D. Jacobson, Optimal stochastic linear systems with exponential performance criteria and their relation to deterministic differential games, IEEE Trans. Auto. Control 18, 124 (1973).
  56. P. Whittle, Risk Sensitive Optimal Control, Wiley-Interscience Series in Systems and Optimization (Wiley, 1990).
  57. J. M. Raser and E. K. O'Shea, Noise in gene expression: Origins, consequences, and control, Science 309, 2010 (2005).
  58. B. van den Broek, W. Wiegerinck, and H. Kappen, Risk sensitive path integral control, in Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence (UAI2010) (2010).
  59. Y. Xiao, M. T. Angulo, S. Lao, S. T. Weiss, and Y.-Y. Liu, An ecological framework to understand the efficacy of fecal microbiota transplantation, Nat. Commun. 11, 3329 (2020).
  60. A. Skwara, K. Gowda, M. Yousef, J. Diaz-Colunga, A. S. Raman, A. Sanchez, M. Tikhonov, and S. Kuehn, Statistically learning the functional landscape of microbial communities, Nat. Ecol. Evol. 7, 1823 (2023).
  61. F. Brauer and C. Castillo-Chavez, Mathematical Models in Population Biology and Epidemiology, Texts in Applied Mathematics, Vol. 40 (Springer, New York, 2012).
  62. R. M. D'Souza, M. Di Bernardo, and Y.-Y. Liu, Controlling complex networks with complex nodes, Nat. Rev. Phys. 5, 250 (2023).
  63. E. Theodorou, J. Buchli, and S. Schaal, Reinforcement learning of motor skills in high dimensions: A path integral approach, in 2010 IEEE International Conference on Robotics and Automation (IEEE, Anchorage, AK, 2010), pp. 2397–2403.
  64. E. Todorov and Weiwei Li, A generalized iterative LQG method for locally-optimal feedback control of constrained nonlinear stochastic systems, in Proceedings of the 2005, American Control Conference, 2005 (IEEE, Portland, OR, 2005), pp. 300–306.
  65. S. Satoh, H. J. Kappen, and M. Saeki, An iterative method for nonlinear stochastic optimal control based on path integrals, IEEE Trans. Auto. Control 62, 262 (2017).
  66. M. Rathinam and M. Yu, State and parameter estimation from exact partial state observation in stochastic reaction networks, J. Chem. Phys. 154, 034103 (2021).
  67. Z. Fang, A. Gupta, and M. Khammash, Stochastic filtering for multiscale stochastic reaction networks based on hybrid approximations, J. Comput. Phys. 467, 111441 (2022).
  68. C. B. Hammouda, M. Chupin, S. Münker, and R. Tempone, Filtered Markovian projection: Dimensionality reduction in filtering for stochastic reaction networks, arXiv:2502.07918.
  69. K. Nakamura and T. J. Kobayashi, Optimal sensing and control of run-and-tumble chemotaxis, Phys. Rev. Res. 4, 013120 (2022).
  70. R. Boel and P. Varaiya, Optimal control of jump processes, SIAM J. Control Optim. 15, 92 (1977).
  71. I. Csiszár, Why least squares and maximum entropy? An axiomatic approach to inference for linear inverse problems, Ann. Stat. 19, 2032 (1991).
  72. S.-I. Amari, α-Divergence is unique, belonging to both f-divergence and Bregman divergence classes, IEEE Trans. Inf. Theor. 55, 4925 (2009).
  73. M. Opper and G. Sanguinetti, Variational inference for Markov jump processes, in Advances in Neural Information Processing Systems, edited by J. Platt, D. Koller, Y. Singer, and S. Roweis (Curran Associates, New York, 2007), Vol. 20.
  74. X. Guo and O. Hernandez-Lerma, Drift and monotonicity conditions for continuous-time controlled Markov chains with an average criterion, IEEE Trans. Auto. Control 48, 236 (2003).
  75. S. R. Pliska, Controlled jump processes, Stochastic Processes Appl. 3, 259 (1975).
  76. R. Rishel, A minimum principle for controlled jump processes, in Control Theory, Numerical Methods and Computer Systems Modelling, edited by A. Bensoussan and J. L. Lions (Springer, Berlin, 1975), pp. 493–508.
  77. M. Davis and R. Elliott, Optimal control of a jump process, Z. verw. Geb. 40, 183 (1977).
  78. S. N. Cohen and R. J. Elliott, Stochastic Calculus and Applications, Probability and Its Applications (Birkhäuser, New York, 2015).
  79. X. Guo, Q. Liu, and T. Zhou, Optimal control of stochastic fluctuations in biochemical reactions, J. Biol. Syst. 17, 283 (2009).
  80. W. Zhang, C. Hartmann, and M. von Kleist, Optimal control of Markov jump processes: Asymptotic analysis, algorithms and applications to the modelling of chemical reaction systems, arXiv:1512.00216.
  81. C. Lefèvre, Optimal control of a birth and death epidemic process, Oper. Res. 29, 971 (1981).
  82. H. Cai and X. Luo, Stochastic control of an epidemic process, Int. J. Syst. Sci. 25, 821 (1994).
  83. M. Feinberg, Foundations of Chemical Reaction Network Theory, Applied Mathematical Sciences, Vol. 202 (Springer, Cham, 2019).
  84. W. H. Fleming, Exit probabilities and optimal stochastic control, Appl. Math. Opt. 4, 329 (1977).
  85. C. J. Holland, A new energy characterization of the smallest eigenvalue of the Schrödinger equation, Commun. Pure Appl. Math. 30, 755 (1977).
  86. W. H. Fleming, Logarithmic transformations and stochastic control, in Advances in Filtering and Optimal Stochastic Control, edited by W. H. Fleming and Luis G. Gorostiza (Springer, Berlin, 1982), pp. 131–141.
  87. W. Feller, An Introduction to Probability Theory and Its Applications, 3rd ed., Vol. 1 (Wiley, 1991).

Outline

Information

PRX Life 3, 033027– Published 26 September, 2025
  • Received 27 September 2024
  • Accepted 18 August 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation