- Physical Review Journals
All JournalsPhysics Magazine

PRX Life

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Pseudo-perplexity in one fell swoop for protein fitness estimation

Pranav Kantroo1,2,*, Günter P. Wagner3,4,5, and Benjamin B. Machta2,6,†

  • 1Computational Biology and Bioinformatics Program, Yale University, New Haven, Connecticut 06520, USA
  • 2Quantitative Biology Institute, Yale University, New Haven, Connecticut 06520, USA
  • 3Department of Ecology and Evolutionary Biology, Yale University, New Haven, Connecticut 06520, USA
  • 4Department of Evolutionary Biology, University of Vienna, Djerassi Platz 1, A-1030 Vienna, Austria
  • 5Hagler Institute for Advanced Studies, Texas A&M, College Station, Texas 77843, USA
  • 6Department of Physics, Yale University, New Haven, Connecticut 06520, USA

  • *Contact author: pranav.kantroo@yale.edu
  • †Contact author: benjamin.machta@yale.edu
PDF

PRX Life 3, 033014 – Published 21 August, 2025

DOI: https://doi.org/10.1103/zhx7-hcmm

Abstract

Protein language models trained on the masked language modeling objective learn to predict the identity of hidden amino acid residues within a sequence using the remaining observable sequence as context. They do so by embedding the residues into a high dimensional space that encapsulates the relevant contextual cues. These embedding vectors serve as an informative context-sensitive representation that not only aids with the defined training objective, but can also be used for other tasks by downstream models. We propose a scheme to use the embeddings of an unmasked sequence to estimate the corresponding masked probability vectors for all the positions in a single forward pass through the language model. This one fell swoop (OFS) approach allows us to efficiently estimate the pseudo-perplexity of the sequence, a measure of the model's uncertainty in its predictions, that can also serve as a fitness estimate. We find that ESM2 OFS pseudo-perplexity performs nearly as well as the true pseudo-perplexity at fitness estimation, and more notably it defines a new state of the art on the ProteinGym Indels benchmark. The strong performance of the fitness measure prompted us to investigate if it could be used to detect the elevated stability reported in reconstructed ancestral sequences. We find that this measure ranks ancestral reconstructions as more fit than extant sequences. Finally, we show that the computational efficiency of the technique allows for the use of Monte Carlo methods that can rapidly explore functional sequence space.

View figure in article

Physics Subject Headings (PhySH)

Collections

This article appears in the following collection:

Machine Learning and Structural Biology

Curated by the organizers of the Machine Learning for Structural Biology Workshop at NeurIPS, this collection features studies at the intersection of computer science and biology, showcasing how machine learning is impacting our understanding of biological molecules and their processes.

Every article published in this collection underwent a rigorous peer review process, adhering to the same high standards applied to all papers handled by the journal. The PRX Life editorial team managed the peer review and made all editorial decisions.

Article Text

References (72)

  1. J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko et al., Highly accurate protein structure prediction with alphafold, Nature (London) 596, 583 (2021).
  2. J. Dauparas, I. Anishchenko, N. Bennett, H. Bai, R. J. Ragotte, L. F. Milles, B. I. M. Wicky, A. Courbet, R. J. de Haas, N. Bethel et al., Robust deep learning–based protein sequence design using ProteinMPNN, Science 378, 49 (2022).
  3. J. L. Watson, D. Juergens, N. R. Bennett, B. L. Trippe, J. Yim, H. E. Eisenach, W. Ahern, A. J. Borst, R. J. Ragotte, L. F. Milles et al., De novo design of protein structure and function with RFdiffusion, Nature (London) 620, 1089 (2023).
  4. P. Notin, M. Dias, J. Frazer, J. Marchena-Hurtado, A. N. Gomez, D. Marks, and Y. Gal, Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval, in Proceedings of the 39th International Conference on Machine Learning, edited by K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato (PMLR, 2022), Vol. 162, pp. 16990–17017.
  5. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, Attention is all you need, Adv. Neural Inf. Proc. Syst. 30 (2017).
  6. Z. Lin, H. Akin, R. Rao, B. Hie, Z. Zhu, W. Lu, N. Smetanin, R. Verkuil, O. Kabeli, Y. Shmueli et al., Evolutionary-scale prediction of atomic-level protein structure with a language model, Science 379, 1123 (2023).
  7. E. Nijkamp, J. A. Ruffolo, E. N. Weinstein, N. Naik, and A. Madani, Progen2: exploring the boundaries of protein language models, Cell Syst. 14, 968 (2023).
  8. R. M. Rao, J. Liu, R. Verkuil, J. Meier, J. Canny, P. Abbeel, T. Sercu, and A. Rives, Msa transformer, in International Conference on Machine Learning (PMLR, 2021), pp. 8844–8856.
  9. C. Hsu, R. Verkuil, J. Liu, Z. Lin, B. Hie, T. Sercu, A. Lerer, and A. Rives, Learning inverse folding from millions of predicted structures, in International Conference on Machine Learning (PMLR, 2022), pp. 8946–8970.
  10. M. Heinzinger, K. Weissenow, J. G. Sanchez, A. Henkel, M. Steinegger, and B. Rost, Bilingual language model for protein sequence and structure, NAR Genomics and Bioinformatics 6, lqae150 (2024).
  11. R. Verkuil, O. Kabeli, Y. Du, B. I. M. Wicky, L. F. Milles, J. Dauparas, D. Baker, S. Ovchinnikov, T. Sercu, and A. Rives, Language models generalize beyond natural proteins, bioRxiv 2022–12 (2022).
  12. T. Bepler and B. Berger, Learning the protein language: evolution, structure, and function, Cell Syst. 12, 654e3 (2021).
  13. J. A. Ruffolo and A. Madani, Designing proteins with language models, Nat. Biotechnol. 42, 200 (2024).
  14. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, in Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) (2019), pp. 4171–4186.
  15. E. C. Alley, G. Khimulya, S. Biswas, M. AlQuraishi, and G. M. Church, Unified rational protein engineering with sequence-based deep representation learning, Nat. Methods 16, 1315 (2019).
  16. A. Elnaggar, M. Heinzinger, C. Dallago, G. Rehawi, Y. Wang, L. Jones, T. Gibbs, T. Feher, C. Angerer, M. Steinegger et al., Prottrans: Toward understanding the language of life through self-supervised learning, IEEE Trans. Pattern Anal. Mach. Intell. 44, 7112 (2021).
  17. A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, Improving language understanding by generative pre-training (2018), https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf.
  18. J.-E. Shin, A. J. Riesselman, A. W. Kollasch, C. McMahon, E. Simon, C. Sander, A. Manglik, A. C. Kruse, and D. S. Marks, Protein design and variant prediction using autoregressive generative models, Nat. Commun. 12, 2403 (2021).
  19. N. Ferruz, S. Schmidt, and B. Höcker, Protgpt2 is a deep unsupervised language model for protein design, Nat. Commun. 13, 4348 (2022).
  20. D. Hesslow, N. Zanichelli, P. Notin, I. Poli, and D. Marks, Rita: A study on scaling up generative protein sequence models, arXiv:2205.05789.
  21. J. Horne and D. Shukla, Recent advances in machine learning variant effect prediction tools for protein engineering, Indust. Eng. Chem. Res. 61, 6235 (2022).
  22. J. Meier, R. Rao, R. Verkuil, J. Liu, T. Sercu, and A. Rives, Language models enable zero-shot prediction of the effects of mutations on protein function, in Advances in Neural Information Processing Systems, edited by M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. W. Vaughan (Curran Associates, Inc., 2021), Vol. 34, pp. 29287–29303.
  23. P. Notin, A. Kollasch, D. Ritter, L. Van Niekerk, S. Paul, H. Spinner, N. Rollins, A. Shaw, R. Orenbuch, R. Weitzman et al., Proteingym: Large-scale benchmarks for protein fitness prediction and design, Adv. Neural Inf. Proc. Syst. 36 (2024).
  24. A. Madani, B. Krause, E. R. Greene, S. Subramanian, B. P. Mohr, J. M. Holton, J. L. Olmos, C. Xiong, Z. Z. Sun, R. Socher et al., Large language models generate functional protein sequences across diverse families, Nat. Biotechnol. 41, 1099 (2023).
  25. J. Salazar, D. Liang, T. Q. Nguyen, and K. Kirchhoff, Masked language model scoring, in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, edited by D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (ACL, 2020), pp. 2699–2712.
  26. M. Steinegger and J. Söding, MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets, Nat. Biotechnol. 35, 1026 (2017).
  27. D. M. Fowler and S. Fields, Deep mutational scanning: A new style of protein science, Nat. Methods 11, 801 (2014).
  28. J. Su, C. Han, Y. Zhou, J. Shan, X. Zhou, and F. Yuan, SaProt: protein language modeling with structure-aware vocabulary (2024), https://openreview.net/forum?id=6MRm3G4NiU.
  29. M. Li, Y. Tan, X. Ma, B. Zhong, H. Yu, Z. Zhou, W. Ouyang, B. Zhou, P. Tan, and L. Hong, Prosst: Protein language modeling with quantized structure and disentangled attention, Advances in Neural Information Processing Systems 37, 35700 (2024).
  30. S. Paul, A. Kollasch, P. Notin, and D. Marks, Combining structure and sequence for superior fitness prediction, in NeurIPS 2023 Generative AI and Biology (GenBio) Workshop (2023).
  31. D. L. Trudeau, M. Kaltenbach, and D. S. Tawfik, On the potential origins of the high stability of reconstructed ancestral proteins, Mol. Biol. Evol. 33, 2633 (2016).
  32. P. Kapli, Z. Yang, and M. J. Telford, Phylogenetic tree building in the genomic age, Nat. Rev. Genet. 21, 428 (2020).
  33. J. B. Joy, R. H. Liang, R. M. McCloskey, T. Nguyen, and A. F. Y. Poon, Ancestral reconstruction, PLoS Comput. Biol. 12, e1004763 (2016).
  34. G. K. A. Hochberg and J. W. Thornton, Reconstructing ancient proteins to understand the causes of structure and function, Annu. Rev. Biophys. 46, 247 (2017).
  35. A. S. Pillai, S. A. Chandler, Y. Liu, A. V. Signore, C. R. Cortez-Romero, J. L. P. Benesch, A. Laganowsky, J. F. Storz, G. K. A. Hochberg, and J. W. Thornton, Origin of complexity in haemoglobin evolution, Nature (London) 581, 480 (2020).
  36. S. F. Field, M. Y. Bulina, I. V. Kelmanson, J. P. Bielawski, and M. V. Matz, Adaptive evolution of multicolored fluorescent proteins in reef-building corals, J. Mol. Evol. 62, 332 (2006).
  37. V. J. Lynch, G. May, and G. P. Wagner, Regulatory evolution through divergence of a phosphoswitch in the transcription factor cebpb, Nature (London) 480, 383 (2011).
  38. M. C. Nnamani, S. Ganguly, E. M. Erkenbrack, V. J. Lynch, L. S. Mizoue, Y. Tong, H. L. Darling, M. Fuxreiter, J. Meiler, and G. P. Wagner, A derived allosteric switch underlies the evolution of conditional cooperativity between hoxa11 and foxo1, Cell Rep. 15, 2097 (2016).
  39. K. J. Brayer, V. J. Lynch, and G. P. Wagner, Evolution of a derived protein–protein interaction between hoxa11 and foxo1a in mammals caused by changes in intramolecular regulation, Proc. Natl. Acad. Sci. USA 108, E414 (2011).
  40. N. M. Hendrikse, A. Sandegren, T. Andersson, J. Blomqvist, Å. Makower, D. Possner, C. Su, N. Thalén, A. Tjernberg, U. Westermark et al., Ancestral lysosomal enzymes with increased activity harbor therapeutic potential for treatment of hunter syndrome, iScience 24, 102154 (2021).
  41. E. A. Gaucher, S. Govindarajan, and O. K. Ganesh, Palaeotemperature trend for precambrian life inferred from resurrected proteins, Nature (London) 451, 704 (2008).
  42. V. A. Risso, J. A. Gavira, D. F. Mejia-Carmona, E. A. Gaucher, and J. M. Sanchez-Ruiz, Hyperstability and substrate promiscuity in laboratory resurrections of precambrian β-lactamases, J. Am. Chem. Soc. 135, 2899 (2013).
  43. V. A. Risso, J. A. Gavira Gallardo, J. M. Sánchez Ruiz et al., Thermostable and promiscuous precambrian proteins (2014).
  44. M. A. Spence, J. A. Kaczmarski, J. W. Saunders, and C. J. Jackson, Ancestral sequence reconstruction for protein engineers, Curr. Opin. Struct. Biol. 69, 131 (2021).
  45. S. R. Johnson, X. Fu, S. Viknander, C. Goldin, S. Monaco, A. Zelezniak, and K. K. Yang, Computational scoring and experimental evaluation of enzymes generated by neural networks, Nat. Biotechnol. 43, 396 (2025).
  46. A. L. Pey, D. Rodriguez-Larrea, S. Bomke, S. Dammers, R. Godoy-Ruiz, M. M. Garcia-Mira, and J. M. Sanchez-Ruiz, Engineering proteins with tunable thermodynamic and kinetic stabilities, Proteins: Struct. Funct. Bioinform. 71, 165 (2008).
  47. B. J. Sullivan, T. Nguyen, V. Durani, D. Mathur, S. Rojas, M. Thomas, T. Syu, and T. J. Magliery, Stabilizing proteins from sequence statistics: the interplay of conservation and correlation in triosephosphate isomerase stability, J. Mol. Biol. 420, 384 (2012).
  48. R. S. Komor, P. A. Romero, C. B. Xie, and F. H. Arnold, Highly thermostable fungal cellobiohydrolase I (Cel7A) engineered using predictive methods, Protein Eng. Des. Select. 25, 827 (2012).
  49. M. Sternke, K. W. Tripp, and D. Barrick, Consensus sequence design as a general strategy to create hyperstable, biologically active proteins, Proc. Natl. Acad. Sci. USA 116, 11275 (2019).
  50. M. Lynch and A. Force, The probability of duplicate gene preservation by subfunctionalization, Genetics 154, 459 (2000).
  51. L. C. Wheeler, S. A. Lim, S. Marqusee, and M. J. Harms, The thermostability and specificity of ancient proteins, Curr. Opin. Struct. Biol. 38, 37 (2016).
  52. J. Huerta-Cepas, F. Serra, and P. Bork, ETE 3: reconstruction, analysis, and visualization of phylogenomic data, Mol. Biol. Evol. 33, 1635 (2016).
  53. P. J. A. Cock, T. Antao, J. T. Chang, B. A. Chapman, C. J. Cox, A. Dalke, I. Friedberg, T. Hamelryck, F. Kauff, B. Wilczynski et al., Biopython: Freely available python tools for computational molecular biology and bioinformatics, Bioinformatics 25, 1422 (2009).
  54. N. Cliff, Dominance statistics: Ordinal analyses to answer ordinal questions. Psych. Bull. 114, 494 (1993).
  55. T. Paysan-Lafosse, M. Blum, S. Chuguransky, T. Grego, B. L. Pinto, G. A. Salazar, M. L. Bileschi, P. Bork, A. Bridge, L. Colwell et al., InterPro in 2022, Nucl. Acids Res. 51, D418 (2023).
  56. J. Mistry, S. Chuguransky, L. Williams, M. Qureshi, G. A. Salazar, E. L. L. Sonnhammer, S. C. E. Tosatto, L. Paladin, S. Raj, L. J. Richardson et al., Pfam: The protein families database in 2021, Nucl. Acids Res. 49, D412 (2021).
  57. B. Q. Minh, M. W. Hahn, and R. Lanfear, New methods to calculate concordance factors for phylogenomic datasets, Mol. Biol. Evol. 37, 2727 (2020).
  58. S. Kalyaanamoorthy, B. Q. Minh, T. K. F. Wong, A. Von Haeseler, and L. S. Jermiin, ModelFinder: Fast model selection for accurate phylogenetic estimates, Nat. Methods 14, 587 (2017).
  59. H. Ashkenazy, O. Penn, A. Doron-Faigenboim, O. Cohen, G. Cannarozzi, O. Zomer, and T. Pupko, FastML: A web server for probabilistic reconstruction of ancestral sequences, Nucl. Acids Res. 40, W580 (2012).
  60. S. Arora and A. Goyal, A theory for emergence of complex skills in language models, arXiv:2307.15936.
  61. P. Notin, N. Rollins, Y. Gal, C. Sander, and D. Marks, Machine learning for functional protein design, Nat. Biotechnol. 42, 216 (2024).
  62. B. Hie, S. Candido, Z. Lin, O. Kabeli, R. Rao, N. Smetanin, T. Sercu, and A. Rives, A high-level programming language for generative protein design, bioRxiv 2022–12 (2022).
  63. K. H. Sumida, R. Núñez-Franco, I. Kalvet, S. J. Pellock, B. I. M. Wicky, L. F. Milles, J. Dauparas, J. Wang, Y. Kipnis, N. Jameson et al., Improving protein expression, stability, and function with ProteinMPNN, J. Am. Chem. Soc. 146, 2054 (2024).
  64. R. W. Shuai, J. A. Ruffolo, and J. J. Gray, Iglm: Infilling language modeling for antibody sequence design, Cell Syst. 14, 979 (2023).
  65. T. Truong Jr. and T. Bepler, PoET: A generative model of protein families as sequences-of-sequences, Advances in Neural Information Processing Systems (Curran Associates, Inc., 2023), Vol. 36, pp. 77379–77415.
  66. D. Sgarbossa, U. Lupo, and A.-F. Bitbol, Generative power of a protein language model trained on multiple sequence alignments, Elife 12, e79854 (2023).
  67. S. Biswas, G. Khimulya, E. C. Alley, K. M. Esvelt, and G. M. Church, Low-n protein engineering with data-efficient deep learning, Nat. Methods 18, 389 (2021).
  68. I. Anishchenko, S. J. Pellock, T. M. Chidyausiku, T. A. Ramelot, S. Ovchinnikov, J. Hao, K. Bafna, C. Norn, A. Kang, A. K. Bera et al., De novo protein design by deep network hallucination, Nature (London) 600, 547 (2021).
  69. C. Zhang, M. Shine, A. M. Pyle, and Y. Zhang, Us-align: universal structure alignments of proteins, nucleic acids, and macromolecular complexes, Nat. Methods 19, 1109 (2022).
  70. G. Hinton, O. Vinyals, and J. Dean, Distilling the knowledge in a neural network, arXiv:1503.02531.
  71. A. Shaw, H. Spinner, J. Shin, S. Gurev, N. Rollins, and D. Marks, Removing bias in sequence models of protein fitness, bioRxiv 2023–09 (2023).
  72. F. Ding and J. Steinhardt, Protein language models are biased by unequal sequence sampling across the tree of life, ICLR 2024 Workshop on Generative and Experimental Perspectives for Biomolecular Design (2024).

Outline

Information

PRX Life 3, 033014– Published 21 August, 2025
  • Received 15 July 2024
  • Accepted 25 July 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation