- Open Access
Intuitive dissection of the Gaussian information bottleneck method with an application to optimal prediction
Phys. Rev. Research 7, 043261 – Published 5 December, 2025
DOI: https://doi.org/10.1103/fnd1-8hty
Abstract
Efficient signal representation is essential for the functioning of living and artificial systems operating under resource constraints. A widely recognized framework for deriving such representations is the information bottleneck method, which yields the optimal strategy for encoding a random variable, such as the signal, in a way that preserves maximal information about a functionally relevant variable, subject to an explicit constraint on the amount of information encoded. While in its general formulation the information bottleneck method is a numerical scheme, it admits an analytical solution in an important special case where the variables involved are jointly Gaussian. In this setting, the solution predicts discrete transitions in the dimensionality of the optimal representation as the encoding capacity is increased. Although these signature transitions, along with other features of the optimal strategy, can be derived from a constrained optimization problem, a clear and intuitive understanding of their emergence is still lacking. In our work, we advance our understanding of the Gaussian information bottleneck method through multiple mutually enriching perspectives, including geometric and information-theoretic ones. These perspectives offer intuition about the set of optimal encoding directions, the nature of the critical points where the optimal number of encoding components changes, and the way the optimal strategy navigates between these critical points. We then apply our treatment of the information bottleneck to a previously studied signal prediction problem, obtaining insights into how different features of the signal are encoded across multiple components to enable optimal prediction of future signals. Altogether, our work deepens the foundational understanding of the information bottleneck method in the Gaussian setting, motivating the exploration of analogous perspectives in broader, non-Gaussian contexts.
Physics Subject Headings (PhySH)
Article Text
Supplemental Material
References (35)
- C. G. Bowsher and P. S. Swain, Environmental sensing, information transfer, and cellular decision-making, Curr. Opin. Biotechnol. 28, 149 (2014).
- J. D. Victor, S. D. Boie, E. G. Connor, J. P. Crimaldi, G. B. Ermentrout, and K. I. Nagel, Olfactory navigation and the receptor nonlinearity, J. Neurosci. 39, 3713 (2019).
- S. E. Palmer, O. Marre, M. J. Berry, and W. Bialek, Predictive information in a sensory population, Proc. Natl. Acad. Sci. USA 112, 6908 (2015).
- E. Nelson, M. Corah, and N. Michael, Environment model adaptation for mobile robot exploration, Auton. Robots 42, 257 (2018).
- A. Goyal, R. Islam, D. J. Strouse, Z. Ahmed, H. Larochelle, M. Botvinick, Y. Bengio, and S. Levine, Infobot: Transfer and exploration via the information bottleneck, in International Conference on Learning Representations (2019).
- N. Tishby, F. C. Pereira, and W. Bialek, The information bottleneck method, in Proceedings of the 37th Allerton Conference on Communication and Computation (University of Illinois, 1999).
- V. Sachdeva, T. Mora, A. M. Walczak, and S. E. Palmer, Optimal prediction with resource constraints using the information bottleneck, PLoS Comput. Biol. 17, e1008743 (2021).
- A. J. Tjalma, V. Galstyan, J. Goedhart, L. Slim, N. B. Becker, and P. R. ten Wolde, Trade-offs between cost and information in cellular prediction, Proc. Natl. Acad. Sci. USA 120, e2303078120 (2023).
- Z. Goldfeld and Y. Polyanskiy, The information bottleneck problem and its applications in machine learning, IEEE J. Sel. Areas Inf. Theory 1, 19 (2020).
- G. Chechik, A. Globerson, N. Tishby, and Y. Weiss, Information bottleneck for Gaussian variables, J. Mach. Learn. Res. 6, 165 (2005).
- W. H. de Ronde, F. Tostevin, and P. R. ten Wolde, Effect of feedback on the fidelity of information transmission of time-varying signals, Phys. Rev. E 82, 031914 (2010).
- M. Chalk, O. Marre, and G. Tkačik, Toward a unified theory of efficient, predictive, and sparse coding, Proc. Natl. Acad. Sci. USA 115, 186 (2018).
- R. E. Kalman, A new approach to linear filtering and prediction problems, J. Basic Eng. 82, 35 (1960).
- A. Scaglione, P. Stoica, S. Barbarossa, G. B. Giannakis, and H. Sampath, Optimal designs for space-time linear precoders and decoders, IEEE Trans. Signal Process. 50, 1051 (2002).
- H. Hotelling, Relations between two sets of variates, Biometrika 28, 321 (1936).
- F. Creutzig, A. Globerson, and N. Tishby, Past-future information bottleneck in dynamical systems, Phys. Rev. E 79, 041925 (2009).
- N. B. Becker, A. Mugler, and P. R. ten Wolde, Optimal prediction by cellular signaling networks, Phys. Rev. Lett. 115, 258103 (2015).
- See Supplemental Material at http://link.aps.org/supplemental/10.1103/fnd1-8hty for details and additional studies, which includes Ref. [34].
- V. Ngampruetikorn and D. J. Schwab, Perturbation theory for the information bottleneck, in Advances in Neural Information Processing Systems (Curran Associates, Inc., 2021), Vol. 34, pp. 21008–21018.
- D. Mulder, P. R. ten Wolde, and T. E. Ouldridge, Exploiting bias in optimal finite-time copying protocols, Phys. Rev. Res. 7, L012026 (2025).
- B. R. Gálvez, R. Thobaben, and M. Skoglund, The convex information bottleneck Lagrangian, Entropy 22, 98 (2020).
- A. Kolchinsky, B. D. Tracey, and D. H. Wolpert, Nonlinear information bottleneck, Entropy 21, 1181 (2019).
- R. M. Hecht, E. Noor, and N. Tishby, Speaker recognition by gaussian information bottleneck, in Proceedings of Interspeech (2009), pp. 1567–1570.
- A. Gordon, A. Banerjee, M. Koch-Janusz, and Z. Ringel, Relevance in the renormalization group and in information theory, Phys. Rev. Lett. 126, 240601 (2021).
- A. G. Kline and S. E. Palmer, Gaussian information bottleneck and the non-perturbative renormalization group, New J. Phys. 24, 033007 (2022).
- M. Rey and V. Roth, Meta-Gaussian information bottleneck, in Advances in Neural Information Processing Systems (2012), Vol. 25.
- M. M. Mahvari, M. Kobayashi, and A. Zaidi, Scalable vector Gaussian information bottleneck, in IEEE International Symposium on Information Theory (ISIT) (Melbourne, 2021).
- V. Ngampruetikorn and D. J. Schwab, Generalized information bottleneck for Gaussian variables, arXiv:2303.17762.
- S. Still, D. A. Sivak, A. J. Bell, and G. E. Crooks, Thermodynamics of prediction, Phys. Rev. Lett. 109, 120604 (2012).
- V. Ngampruetikorn and D. J. Schwab, Information bottleneck theory of high-dimensional regression: Relevancy, efficiency and optimality, in Advances in Neural Information Processing Systems (2022), Vol. 35, pp. 9784–9796.
- M. Bauer and W. Bialek, Information bottleneck in molecular sensing, PRX Life 1, 023005 (2023).
- T. Wu and I. Fischer, Phase transitions for the information bottleneck in representation learning, in International Conference on Learning Representations (2020).
- M. Reinhardt, G. Tkačik, and P. R. ten Wolde, Path weight sampling: Exact Monte Carlo computation of the mutual information between stochastic trajectories, Phys. Rev. X 13, 041017 (2023).
- S. Särkkä and A. Solin, Applied Stochastic Differential Equations, Institute of Mathematical Statistics Textbooks (Cambridge University Press, 2019).
- V. Galstyan, Jupyter notebooks for reproducing the figures of the paper titled: “Intuitive dissection of the Gaussian information bottleneck method with an application to optimal prediction” (https://doi.org/10.1103/fnd1-8hty), Zenodo (2025), available at: https://doi.Org/10.5281/zenodo.17495525.
