- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Phylogenetic corrections and higher-order sequence statistics in protein families: Potts vs multiple sequence alignment transformer machine learning models

Kisan Khatri1, Ronald M. Levy2, and Allan Haldane1,*

  • 1Department of Physics and Center for Biophysics and Computational Biology, Temple University, Philadelphia, Pennsylvania 19122, USA
  • 2Department of Chemistry and Center for Biophysics and Computational Biology, Temple University, Philadelphia, Pennsylvania 19122, USA

  • *Contact author: allan.haldane@temple.edu
PDF

Phys. Rev. Research 7, 043077 – Published 20 October, 2025

DOI: https://doi.org/10.1103/g5cx-1vhj

Abstract

Recent generative machine learning models applied to protein multiple sequence alignment (MSA) datasets include simple and interpretable physics-based Potts covariation models and other machine learning models such as MSA Transformer (MSA-T). The best models accurately reproduce MSA statistics induced by the biophysical constraints within proteins, raising the question of which functional forms best model the underlying physics. The Potts model is usually specified by an effective potential including pairwise residue-residue interaction terms, but it has been suggested that MSA-T can capture the effects induced by effective potentials that include more than pairwise interactions and implicitly account for phylogenetic structure in the MSA. Here, we compare the ability of the Potts model and MSA-T to reconstruct higher-order sequence statistics reflecting complex biophysical sequence constraints. We show how the model performance depends greatly on the treatment of phylogenetic relationships between the sequences, which can induce nonbiophysical mutational covariation in MSAs. When using explicit corrections for phylogenetic dependencies, we find that the Potts model outperforms MSA-T in detecting epistatic interactions of biophysical origin. Furthermore, the sequences generated by the models we studied here are predicted to fold into nativelike structures, as indicated by AlphaFold predictions.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (56)

  1. R. M. Levy, A. Haldane, and W. F. Flynn, Potts Hamiltonian models of protein co-variation, free energy landscapes, and evolutionary fitness, Curr. Opin. Struct. Biol. 43, 55 (2017).
  2. T. Bepler and B. Berger, Learning the protein language: Evolution, structure, and function, Cell Syst. 12, 654 (2021).
  3. D. De Juan, F. Pazos, and A. Valencia, Emerging methods in protein co-evolution, Nat. Rev. Genet. 14, 249 (2013).
  4. M. Weigt, R. A. White, H. Szurmant, J. A. Hoch, and T. Hwa, Identification of direct residue contacts in protein–protein interaction by message passing, Proc. Natl. Acad. Sci. USA 106, 67 (2009).
  5. F. Morcos, A. Pagnani, B. Lunt, A. Bertolino, D. S. Marks, C. Sander, R. Zecchina, J. N. Onuchic, T. Hwa, and M. Weigt, Direct-coupling analysis of residue coevolution captures native contacts across many protein families, Proc. Natl. Acad. Sci. USA 108, E1293 (2011).
  6. T. Mora and W. Bialek, Are biological systems poised at criticality? J. Stat. Phys. 144, 268 (2011).
  7. A. Haldane, W. F. Flynn, P. He, R. Vijayan, and R. M. Levy, Structural propensities of kinase family proteins from a Potts model of residue co-variation, Protein Sci. 25, 1378 (2016).
  8. A. Haldane and R. M. Levy, Influence of multiple-sequence-alignment depth on Potts statistical models of protein covariation, Phys. Rev. E 99, 032405 (2019).
  9. J. Trinquier, G. Uguzzoni, A. Pagnani, F. Zamponi, and M. Weigt, Efficient generative modeling of protein sequences using simple autoregressive models, Nat. Commun. 12, 5800 (2021).
  10. R. M. Rao, J. Liu, R. Verkuil, J. Meier, J. Canny, P. Abbeel, T. Sercu, and A. Rives, MSA transformer, in International Conference on Machine Learning, Proceedings of Machine Learning Research (PMLR, 2021), pp. 8844–8856.
  11. U. Lupo, D. Sgarbossa, and A. F. Bitbol, Protein language models trained on multiple sequence alignments learn phylogenetic relationships, Nat. Commun. 13, 6298 (2022).
  12. F. McGee, S. Hauri, Q. Novinger, S. Vucetic, R. M. Levy, V. Carnevale, and A. Haldane, The generative capacity of probabilistic protein sequence models, Nat. Commun. 12, 6302 (2021).
  13. J. Martin, M. Lequerica Mateos, J. N. Onuchic, I. Coluzza, and F. Morcos, Machine learning in biological physics: From biomolecular prediction to design, Proc. Natl. Acad. Sci. USA 121, e2311807121 (2024).
  14. Y. LeCun, Y. Bengio, and G. Hinton, Deep learning, Nature (London) 521, 436 (2015).
  15. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, Attention is all you need, Advances in Neural Information Processing Systems, edited by I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett 30 (Curran Associates, Inc., Long Beach, California, USA, 2017).
  16. A. Hawkins-Hooker, D. T. Jones, and B. Paige, MSA-conditioned generative protein language models for fitness landscape modelling and design, in Machine Learning for Structural Biology Workshop, NeurIPS (2021).
  17. Y. Hong, J. Song, J. Ko, J. Lee, and W.-H. Shin, S-Pred: Protein structural property prediction using MSA Transformer, Sci. Rep. 12, 13891 (2022).
  18. C. Ma, H. Zhao, L. Zheng, J. Xin, Q. Li, L. Wu, Z. Deng, Y. Lu, Q. Liu, and L. Kong, Retrieved sequence augmentation for protein representation learning, arXiv:2302.12563.
  19. B. Almalki and L. Liao, Tmhc-msat: Accurate prediction of inter-helical residue contacts in transmembrane proteins using MSA Transformer, in Proceedings of the 16th International Conference on Bioinformatics and Computational Biology (BICOB-2024), edited by H. Al-Mubaid, T. Aldwairi, and O. Eulenstein, EPiC Series in Computing (EasyChair, New Orleans, Louisiana, USA, 2024), Vol. 101, pp. 1–10.
  20. F. Cuturello, M. Celoria, A. Ansuini, and A. Cazzaniga, Enhancing predictions of protein stability changes induced by single mutations using MSA-based language models, Bioinformatics 40, btae447 (2024).
  21. R. Chen, G. Foley, and M. Boden, Learning the language of phylogeny with MSA transformer, bioRxiv (2024), doi: https://doi.org/10.1101/2024.12.18.629037.
  22. Y. Chen, G. Chen, and C. Y.-C. Chen, MFTrans: A multi-feature transformer network for protein secondary structure prediction, Int. J. Biol. Macromol. 267, 131311 (2024).
  23. D. Sgarbossa, U. Lupo, and A. F. Bitbol, Generative power of a protein language model trained on multiple sequence alignments, eLife 12, e79854 (2023).
  24. A. Haldane and R. M. Levy, Mi3-GPU: Mcmc-based inverse ising inference on gpus for protein covariation analysis, Comput. Phys. Commun. 260, 107312 (2021).
  25. See Supplemental Material at http://link.aps.org/supplemental/10.1103/g5cx-1vhj for additional figures and analyses supporting the main text, which also includes Refs. [52, 53].
  26. B. Rost, Protein structures sustain evolutionary drift, Folding Des. 2, S19 (1997).
  27. B. Rost, Twilight zone of protein sequence alignments, Protein Engineering 12, 85 (1999).
  28. R. D. Finn, A. Bateman, J. Clements, P. Coggill, R. Y. Eberhardt, S. R. Eddy, A. Heger, K. Hetherington, L. Holm, J. Mistry et al., Pfam: The protein families database, Nucleic Acids Res. 42, D222 (2014).
  29. T. D. Schneider and R. M. Stephens, Sequence logos: A new way to display consensus sequences, Nucleic Acids Res. 18, 6097 (1990).
  30. G. E. Crooks, G. Hon, J.-M. Chandonia, and S. E. Brenner, WebLogo: A sequence logo generator, Genome Res. 14, 1188 (2004).
  31. A. S. Lapedes, B. G. Giraud, L. Liu, and G. D. Stormo, Correlated mutations in models of protein sequences: Phylogenetic and structural effects, Stat. Mol. Biol. Genet. 33, 236 (1999).
  32. J. Y. Dutheil, Detecting coevolving positions in a molecule: Why and how to account for phylogeny, Brief. Bioinform. 13, 228 (2012).
  33. R. Rao, J. Meier, T. Sercu, S. Ovchinnikov, and A. Rives, Transformer protein language models are unsupervised structure learners, in International Conference on Learning Representations (2021), doi: https://doi.org/10.1101/2020.12.15.422761.
  34. A. Colavin, E. Atolia, A. F. Bitbol, and K. C. Huang, Extracting phylogenetic dimensions of coevolution reveals hidden functional signals, Sci. Rep. 12, 820 (2022).
  35. S. D. Dunn, L. M. Wahl, and G. B. Gloor, Mutual information without the influence of phylogeny or entropy dramatically improves residue contact prediction, Bioinformatics 24, 333 (2008).
  36. C. M. Buslje, J. Santos, J. M. Delfino, and M. Nielsen, Correction for phylogeny, small number of observations and data redundancy improves the identification of coevolving amino acid pairs using mutual information, Bioinformatics 25, 1125 (2009).
  37. R. D. Finn, J. Clements, and S. R. Eddy, HMMER web server: Interactive sequence similarity searching, Nucleic Acids Res. 39, W29 (2011).
  38. S. Henikoff and J. G. Henikoff, Amino acid substitution matrices from protein blocks., Proc. Natl. Acad. Sci. USA 89, 10915 (1992).
  39. C. Qin and L. J. Colwell, Power law tails in phylogenetic systems, Proc. Natl. Acad. Sci. USA 115, 690 (2018).
  40. E. R. Horta1 and M. Weigt, On the effect of phylogenetic correlations in coevolution-based contact prediction in proteins, PLoS Comput. Biol. 17, 1 (2021).
  41. N. Bhattacharya, N. Thomas, R. Rao, J. Dauparas, P. K. Koo, D. Baker, Y. S. Song, and S. Ovchinnikov, Interpreting Potts and transformer protein models through the lens of simplified attention, in Pacific Symposium on Biocomputing 2022 (World Scientific, 2021), pp. 34–45.
  42. Y. Hong, J. Lee, and J. Ko, A-Prot: Protein structure modeling using MSA transformer, BMC Bioinf. 23, 93 (2022).
  43. S. Cocco, L. Posani, and R. Monasson, Functional effects of mutations in proteins can be predicted and interpreted by guided selection of sequence covariation information, Proc. Natl. Acad. Sci. USA 121, e2312335121 (2024).
  44. S. Alvarez, C. M. Nartey, N. Mercado, J. A. de la Paz, T. Huseinbegovic, and F. Morcos, In vivo functional phenotypes from a computational epistatic model of evolution, Proc. Natl. Acad. Sci. USA 121, e2308895121 (2024).
  45. B. Zhou, L. Zheng, B. Wu, K. Yi, B. Zhong, Y. Tan, Q. Liu, P. Liò, and L. Hong, A conditional protein diffusion model generates artificial programmable endonuclease sequences with enhanced activity, Cell Discov. 10, 95 (2024).
  46. A. Madani, B. Krause, E. R. Greene, S. Subramanian, B. P. Mohr, J. M. Holton, J. L. Olmos Jr, C. Xiong, Z. Z. Sun, R. Socher et al., Large language models generate functional protein sequences across diverse families, Nat. Biotechnol. 41, 1099 (2023).
  47. Z. Wu, K. E. Johnston, F. H. Arnold, and K. K. Yang, Protein sequence design with deep generative models, Curr. Opin. Chem. Biol. 65, 18 (2021).
  48. M. Mirdita, K. Schütze, Y. Moriwaki, L. Heo, S. Ovchinnikov, and M. Steinegger, ColabFold: Making protein folding accessible to all, Nat. Methods 19, 679 (2022).
  49. S. El-Gebali, J. Mistry, A. Bateman, S. R. Eddy, A. Luciani, S. C. Potter, M. Qureshi, L. J. Richardson, G. A. Salazar, A. Smart et al., The Pfam protein families database in 2019, Nucleic Acids Res. 47, D427 (2019).
  50. https://github.com/ahaldane/Mi3-GPU.
  51. https://github.com/Bitbol-Lab./Iterative_masking.
  52. https://doi.org/10.5281/zenodo.7684052.
  53. K. Khatri, R. M. Levy, and A. Haldane, Phylogenetic corrections and higher-order sequence statistics in protein families: The Potts vs multiple sequence alignment transformer machine learning models, Zenodo (2025), https://doi.org/10.5281/zenodo.15876209
  54. K. Khatri, Potts vs MSA Transformer r20 analysis, 2025, https://github.com/kisan-khatri/Potts_vs_MSA_Transformer-r20_Analysis.
  55. M. Remmert, A. Biegert, A. Hauser, and J. Söding, HHBlits: Lightning-fast iterative protein sequence searching by HMM-HMM alignment, Nat. Methods 9, 173 (2012).
  56. M. Mirdita, L. von den Driesch, C. Galiez, M. J. Martin, J. Söding, and M. Steinegger, Uniclust databases of clustered and deeply annotated protein sequences and alignments, Nucleic Acids Res. 45, D170 (2017).

Outline

Information

Phys. Rev. Research 7, 043077– Published 20 October, 2025
  • Received 28 February 2025
  • Accepted 7 September 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation