- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Letter
  • Open Access

Selecting simple, transferable models with the supremum principle

Cody Petrie*, Christian Anderson, Casie Maekawa, Travis Maekawa, and Mark K. Transtrum†

  • Department of Physics and Astronomy, Brigham Young University, Provo, Utah 84602, USA

  • *codypetrie89@gmail.com
  • †mktranstrum@byu.edu
PDF

Phys. Rev. Research 4, L032044 – Published 23 September, 2022

DOI: https://doi.org/10.1103/PhysRevResearch.4.L032044

Abstract

We consider how mathematical models enable predictions for conditions that are qualitatively different from the training data. We propose techniques based on information topology to find models that can apply their learning in regimes for which there is no data. The first step is to use the manifold boundary approximation method to construct simple, reduced models of target phenomena in a data-driven way. We consider the set of all such reduced models and use the topological relationships among them to reason about model selection for new, unobserved phenomena. Given minimal models for several target behaviors, we introduce the supremum principle as a criterion for selecting a new, transferable model. The supremal model, i.e., the least upper bound, is the simplest model that reduces to each of the target behaviors. We illustrate how to discover supremal models with several examples; in each case, the supremal model unifies causal mechanisms to transfer successfully to new target domains. We use these examples to motivate a general algorithm that has formal connections to theories of analogical reasoning in cognitive psychology.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (55)

  1. J. Baxter, A model of inductive bias learning, J. Artif. Intell. Res. 12, 149 (2000).
  2. K. Weiss, T. M. Khoshgoftaar, and D. Wang, A survey of transfer learning, J. Big Data 3, 9 (2016).
  3. R. W. Batterman, The Devil in the Details: Asymptotic Reasoning in Explanation, Reduction, and Emergence (Oxford University Press, New York, 2002).
  4. K. S. Brown and J. P. Sethna, Statistical mechanical approaches to models with many poorly known parameters, Phys. Rev. E 68, 021904 (2003).
  5. K. S. Brown, C. C. Hill, G. A. Calero, C. R. Myers, K. H. Lee, J. P. Sethna, and R. A. Cerione, The statistical mechanics of complex signaling oetworks: Nerve growth factor signaling, Phys. Biol. 1, 184 (2004).
  6. J. J. Waterfall, F. P. Casey, R. N. Gutenkunst, K. S. Brown, C. R. Myers, P. W. Brouwer, V. Elser, and J. P. Sethna, Sloppy-Model Universality Class and the Vandermonde Matrix, Phys. Rev. Lett. 97, 150601 (2006).
  7. B. B. Machta, R. Chachra, M. K. Transtrum, and J. P. Sethna, Parameter space compression underlies emergent theories and predictive models, Science 342, 604 (2013).
  8. S.-i. Amari and H. Nagaoka, Methods of Information Geometry, Vol. 191 (American Mathematical Society, Providence, RI, 2007).
  9. M. K. Transtrum, B. B. Machta, and J. P. Sethna, Why are Nonlinear Fits to Data so Challenging?, Phys. Rev. Lett. 104, 060201 (2010).
  10. C. H. LaMont and P. A. Wiggins, Correspondence between thermodynamics and inference, Phys. Rev. E 99, 052140 (2019).
  11. K. N. Quinn, H. Wilber, A. Townsend, and J. P. Sethna, Chebyshev Approximation and the Global Geometry of Model Predictions, Phys. Rev. Lett. 122, 158302 (2019).
  12. M. K. Transtrum, B. B. Machta, and J. P. Sethna, Geometry of nonlinear least squares with applications to sloppy models and optimization, Phys. Rev. E 83, 036701 (2011).
  13. B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman, Building machines that learn and think like people, Behav. Brain Sci. 40, e253 (2017).
  14. T. Webb, Z. Dulberg, S. Frankland, A. Petrov, R. O'Reilly, and J. Cohen, Learning representations that support extrapolation, in Proceedings of the 37th International Conference on Machine Learning, edited by H. Daumé III and A. Singh, Proceedings of Machine Learning Research, Vol. 119 (PMLR, 2020), pp. 10136–10146.
  15. A. F. Brouwer and M. C. Eisenberg, The underlying connections between identifiability, active subspaces, and parameter space dimension reduction, arXiv:1802.05641.
  16. M. K. Transtrum and P. Qiu, Model Reduction by Manifold Boundaries, Phys. Rev. Lett. 113, 098701 (2014).
  17. M. K. Transtrum, B. B. Machta, K. S. Brown, B. C. Daniels, C. R. Myers, and J. P. Sethna, Perspective: Sloppiness and emergent theories in physics, biology, and beyond, J. Chem. Phys. 143, 010901 (2015).
  18. Although this approximation is constructed by taking γ→0, it does not require the “true” value of γ to be small. Rather, the role of the recovery mechanism can be compressed into a simpler model with an effective infection rate, similar to the effective electron mass in a condensed matter system.
  19. L. Goentoro and M. W. Kirschner, Evidence that fold-change, and not absolute level, of beta-catenin dictates Wnt signaling, Mol. Cell 36, 872 (2009).
  20. P. B. Jensen, L. Pedersen, S. Krishna, and M. H. Jensen, A Wnt oscillator model for somitogenesis, Biophys. J. 98, 943 (2010).
  21. I. H. Riedel-Kruse, C. Müller, and A. C. Oates, Synchrony dynamics during initiation, failure, and rescue of the segmentation clock, Science 317, 1911 (2007).
  22. S. Gibb, A. Zagorska, K. Melton, G. Tenin, I. Vacca, P. Trainor, M. Maroto, and J. K. Dale, Interfering with Wnt signalling alters the periodicity of the segmentation clock, Dev. Biol. 330, 21 (2009).
  23. A. Goldbeter and O. Pourquié, Modeling the segmentation clock as a network of coupled oscillations in the Notch, Wnt and FGF signaling pathways, J. Theor. Biol. 252, 574 (2008).
  24. C. Gomez, E. M. Özbudak, J. Wunderlich, D. Baumann, J. Lewis, and O. Pourquié, Control of segment number in vertebrate embryos, Nature (London) 454, 335 (2008).
  25. Y. Rui, Z. Xu, B. Xiong, Y. Cao, S. Lin, M. Zhang, S. C. Chan, W. Luo, Y. Han, Z. Lu, Z. Ye, H. M. Zhou, J. Han, A. Meng, and S. C. Lin, A β-catenin-independent dorsalization pathway activated by Axin/JNK signaling and antagonized by Aida, Dev. Cell 13, 268 (2007).
  26. M. K. Transtrum, G. Hart, and P. Qiu, Information topology identifies emergent model classes, CoRR, arXiv:1409.6203.
  27. V. G. Švenda, M. K. Transtrum, B. L. Francis, A. T. Sarić, and A. M. Stanković, State estimation model reduction through the manifold boundary approximation method, IEEE Trans. Power Syst. 37, 272 (2022).
  28. A. T. Sarić, A. A. Sarić, M. K. Transtrum, and A. M. Stanković, Symbolic regression for data-driven dynamic model refinement in power systems, IEEE Trans. Power Syst. 36, 2390 (2020).
  29. B. L. Francis, J. R. Nuttall, M. K. Transtrum, A. T. Sarić, and A. M. Stanković, Network reduction in transient stability models using partial response matching, in 2019 North American Power Symposium (NAPS) (IEEE, New York, 2019), pp. 1–6.
  30. M. K. Transtrum, A. T. Sarić, and A. M. Stanković, Measurement-directed reduction of dynamic models in power systems, IEEE Trans. Power Syst. 32, 2243 (2016).
  31. J. E. Jeong, Q. Zhuang, M. K. Transtrum, E. Zhou, and P. Qiu, Experimental design and model reduction in systems biology, Quant. Biol. 6, 287 (2018).
  32. M. K. Transtrum and P. Qiu, Bridging mechanistic and phenomenological models of complex biological systems, PLoS Comput. Biol. 12, e1004915 (2016).
  33. B. K. Mannakee, A. P. Ragsdale, M. K. Transtrum, and R. N. Gutenkunst, Sloppiness and the geometry of parameter space, in Uncertainty in Biology (Springer, Berlin, 2016), pp. 271–299.
  34. Y. Kurniawan, C. L. Petrie, K. J. Williams, M. K. Transtrum, E. B. Tadmor, R. S. Elliott, D. S. Karls, and M. Wen, Bayesian, frequentist, and information geometry approaches to parametric uncertainty quantification of classical empirical interatomic potentials, J. Chem. Phys. 156, 214103 (2022).
  35. G. L. Marschmann, H. Pagel, P. Kügler, and T. Streck, Equifinality, sloppiness, and emergent structures of mechanistic soil biogeochemical models, Envir. Modell. Software 122, 104518 (2019).
  36. T. Nikšić, M. Imbrišak, and D. Vretenar, “Sloppy” nuclear energy density functionals. II. Finite nuclei, Phys. Rev. C 95, 054304 (2017).
  37. J. Rasband, Two reduced models of nerve behavior, Bachelor's thesis, Brigham Young University, 2021.
  38. P. E. Paré, D. Grimsman, A. T. Wilson, M. K. Transtrum, and S. Warnick, Model boundary approximation method as a unifying framework for balanced truncation and singular perturbation approximation, IEEE Trans. Autom. Control 64, 4796 (2019).
  39. T. Gerach, D. Weiß, O. Dössel, and A. Loewe, Observation guided systematic reduction of a detailed human ventricular cell model, in 2019 Computing in Cardiology (CinC) (IEEE, New York, 2019), p. 1.
  40. D. M. Lombardo and W.-J. Rappel, Systematic reduction of a detailed atrial myocyte model, Chaos 27, 093914 (2017).
  41. P. E. Paré, A. T. Wilson, M. K. Transtrum, and S. C. Warnick, A unified view of balanced truncation and singular perturbation approximations, in 2015 American Control Conference (IEEE, New York, 2015), pp. 1989–1994.
  42. See Supplemental Material at http://link.aps.org/supplemental/10.1103/PhysRevResearch.4.L032044 for model equations and more detailed descriptions and examples of the supremum algorithm.
  43. R. Nusse and H. Clevers, Wnt/β-catenin signaling, disease, and emerging therapeutic modalities, Cell 169, 985 (2017).
  44. C. Y. Logan and R. Nusse, The Wnt signaling pathway in development and disease, Annu. Rev. Cell Dev. Biol. 20, 781 (2004).
  45. Y. Ding, S. Su, W. Tang, X. Zhang, S. Chen, G. Zhu, J. Liang, W. Wei, Y. Guo, L. Liu, Y.-G. Chen, and W. Wu, Enrichment of the β-catenin–TCF complex at the S and G2 phases ensures cell survival and cell cycle progression, J. Cell Sci. 127, 4833 (2014).
  46. E. Lee, A. Salic, R. Krger, R. Heinrich, and M. W. Kirschner, The roles of APC and Axin derived from experimental and rheoretical analysis of the Wnt pathway, PLoS Biol. 1, e10 (2003).
  47. L. Ji, B. Lu, R. Zamponi, O. Charlat, R. Aversa, Z. Yang, F. Sigoillot, X. Zhu, T. Hu, J. S. Reece-Hoyes, C. Russ, G. Michaud, J. S. Tchorz, X. Jiang, and F. Cong, USP7 inhibits Wnt/β-catenin signaling through promoting stabilization of Axin, Nat. Commun. 10, 4184 (2019).
  48. R. N. Gutenkunst, J. J. Waterfall, F. P. Casey, K. S. Brown, C. R. Myers, and J. P. Sethna, Universally sloppy parameter sensitivities in systems biology models, PLoS Comput. Biol. 3, e189 (2007).
  49. O. Pourquie, The segmentation clock: Converting embryonic time into spatial pattern, Science 301, 328 (2003).
  50. R. Caruana, Multitask learning, Mach. Learn. 28, 41 (1997).
  51. D. E. Rumelhart and A. A. Abrahamson, A model for analogical reasoning, Cogn. Psychol. 5, 1 (1973).
  52. J. C. Peterson, D. Chen, and T. L. Griffiths, Parallelograms revisited: Exploring the limitations of vector space models for simple analogies, Cognition 205, 104440 (2020).
  53. C. Musto, Enhanced vector space models for content-based recommender systems, in Proceedings of the Fourth ACM Conference on Recommender Systems (ACM Press, New York, 2010), pp. 361–364.
  54. J. N. Reid and A. N. Katz, Vector space applications in metaphor comprehension, Metaphor Symb. 33, 280 (2018).
  55. D. Gentner, Structure-mapping: A theoretical framework for analogy, Cogn. Sci. 7, 155 (1983).

Outline

Information

Phys. Rev. Research 4, L032044– Published 23 September, 2022
  • Received 23 September 2021
  • Revised 11 February 2022
  • Accepted 1 June 2022
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation