- Physical Review Journals
All JournalsPhysics Magazine

PRX Intelligence

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Sampling Weights at Intermediate Temperatures is Optimal for Training Large Language Protein Models

L. Ghiringhelli

A. Zambon* and G. Tiana†

  • *Contact author: alessandro.zambon@unimi.it
  • †Contact author: guido.tiana@unimi.it
PDF

PRX Intelligence 1, 023003 – Published 14 October, 2026

DOI: https://doi.org/10.1103/xvjl-qrwc

Abstract

Using a statistical mechanics framework, we investigate the parameter space of transformer models trained on protein sequence data. We sample the loss landscape at varying temperatures using Langevin dynamics to characterize the low-loss manifold and to understand the mechanisms underlying transformers’ superior performance in protein structure prediction. We find that, at variance with networks not based on the attention mechanism, the lack of a first-order-like transition in the loss of the transformer produces a range of intermediate temperatures with good learning properties; this is true for both synthetic and natural protein sequences. We also show that the parameters of most layers are highly conserved at these temperatures if the dimension of the embedding is optimal, and we provide an operative way to find this dimension. Additionally, we show that the attention matrix is more predictive of the contact maps of the protein at higher temperatures and for higher dimensions of the embedding than those optimal for learning. Finally, we showed that the models sampled at intermediate temperatures can predict the free-energy variation upon mutation, better than models obtained through standard optimization techniques.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (35)

  1. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, Attention is all you need, arXiv:1706.03762.
  2. J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko, et al., Highly accurate protein structure prediction with AlphaFold, Nature (London) 596, 583 (2021).
  3. Z. Lin, H. Akin, R. Rao, B. Hie, Z. Zhu, W. Lu, N. Smetanin, R. Verkuil, O. Kabeli, Y. Shmueli, A. D. Santos Costa, M. Fazel-Zarandi, T. Sercu, S. Candido, and A. Rives, Evolutionary-scale prediction of atomic level protein structure with a language model, bioRxiv (2022).
  4. H. S. Seung, H. Sompolinsky, and N. Tishby, Statistical mechanics of learning from examples, Phys. Rev. A 45, 6056 (1992).
  5. R. Monasson and D. O’Kane, Domains of solutions and replica symmetry breaking in multilayer neural networks, Europhys. Lett. 27, 85 (1994).
  6. Y. Bahri, J. Kadmon, J. Pennington, S. S. Schoenholz, J. Sohl-Dickstein, and S. Ganguli, Statistical mechanics of deep learning, Annu. Rev. Condens. Matter Phys. 11, 501 (2020).
  7. C. Baldassi, A. Ingrosso, C. Lucibello, L. Saglietti, and R. Zecchina, Subdominant dense clusters allow for simple learning and high computational performance in neural networks with discrete synapses, Phys. Rev. Lett. 115, 128101 (2015).
  8. C. Baldassi, C. Borgs, J. T. Chayes, A. Ingrosso, C. Lucibello, L. Saglietti, and R. Zecchina, Unreasonable effectiveness of learning neural networks: From accessible states and robust ensembles to basic algorithmic schemes, Proc. Natl. Acad. Sci. USA 113, E7655 (2016).
  9. J. W. Rocks and P. Mehta, Memorizing without overfitting: Bias, variance, and interpolation in overparameterized models, Phys. Rev. Res. 4, 013201 (2022).
  10. S. Becker, Y. Zhang, and A. A. Lee, Geometry of energy landscapes and the optimizability of deep neural networks, Phys. Rev. Lett. 124, 108301 (2020).
  11. A. Zambon, E. M. Malatesta, G. Tiana, and R. Zecchina, Sampling the space of solutions of an artificial neural network, Phys. Rev. E 112, 045303 (2025).
  12. L. Liu, X. Liu, J. Gao, W. Chen, and J. Han, Understanding the difficulty of training transformers, arXiv:2004.08249.
  13. Y. Tian, Y. Wang, B. Chen, and S. Du, Scan and snap: Understanding training dynamics and token composition in 1-layer transformer, arXiv:2305.16380.
  14. E. Nichani, A. Damian, and J. D. Lee, How transformers learn causal structure with gradient descent, arXiv:2402.14735.
  15. R. Rende, F. Gerace, A. Laio, and S. Goldt, Mapping of attention mechanisms to a generalized Potts model, Phys. Rev. Res. 6, 023057 (2024).
  16. F. Morcos, A. Pagnani, B. Lunt, A. Bertolino, D. S. Marks, C. Sander, R. Zecchina, J. N. Onuchic, T. Hwa, and M. Weigt, Direct-coupling analysis of residue coevolution captures native contacts across many protein families, Proc. Natl. Acad. Sci. USA 108, E1293 (2011).
  17. A. Zambon, F. Caruso, R. Zecchina, and G. Tiana, Controlled Langevin dynamics for sampling of feedforward neural networks trained with minibatches, arXiv:2603.15367.
  18. B. B. Kragelund, C. V. Robinson, J. Knudsen, C. M. Dobson, and F. M. Poulsen, Folding of a four-helix bundle: Studies of acyl-coenzyme a binding protein, Biochemistry 34, 7217 (1995).
  19. G. Franco, M. Cagiada, G. Bussi, and G. Tiana, Statistical mechanical properties of sequence space determine the efficiency of the various algorithms to predict interaction energies and native contacts from protein coevolution, Phys. Biol. 16, 046007 (2019).
  20. G. Tiana and L. Sutto, Equilibrium properties of realistic random heteropolymers and their relevance for globular and naturally unfolded proteins., Phys. Rev. E 84, 061910 (2011).
  21. M. Punta, P. C. Coggill, R. Y. Eberhardt, J. Mistry, J. Tate, C. Boursnell, N. Pang, K. Forslund, G. Ceric, J. Clements, A. Heger, L. Holm, E. L. L. Sonnhammer, S. R. Eddy, A. Bateman, and R. D. Finn, The Pfam protein families database, Nucl. Acids Res. 40, D290 (2012).
  22. R. Rao, J. Meier, T. Sercu, S. Ovchinnikov, and A. Rives, Transformer protein language models are unsupervised structure learners, bioRxiv (2020).
  23. R. Rao, J. Liu, R. Verkuil, J. Meier, J. F. Canny, P. Abbeel, T. Sercu, and A. Rives, MSA Transformer, in Proceedings of the 38th International Conference on Machine Learning, Vol. 139 (PMLR, 2021).
  24. J. Vig, A. Madani, L. R. Varshney, C. Xiong, R. Socher, and N. F. Rajani, BERTology meets biology: Interpreting attention in protein language models, in ICLR 2021 Conference (ICLR, 2021), pp. 1–24.
  25. L. J. McGuffin, K. Bryson, and D. T. Jones, The PSIPRED protein structure prediction server, Bioinformatics 16, 404 (2000).
  26. M.-H. Yu, J. S. Weissman, and P. S. Kim, Contribution of individual side-chains to the stability of BPTI examined by alanine-scanning mutagenesis, J. Mol. Biol. 249, 388 (1995).
  27. V. P. Grantcharova, D. S. Riddle, J. V. Santiago, and D. Baker, Important role of hydrogen bonds in the structurally polarized transition state for folding of the src SH3 domain, Nat. Struct. Mol. Biol. 5, 714 (1998).
  28. E. Cota, S. J. Hamill, S. B. Fowler, and J. Clarke, Two proteins with the same structure respond very differently to mutation: The role of plasticity in protein stability, J. Mol. Biol. 302, 713 (2000).
  29. F. M. Poulsen, B. B. Kragelund, P. Osmark, T. B. Neergaard, J. Schiødt, K. Kristiansen, and J. Knudsen, The formation of a native-like structure containing eight conserved hydrophobic residues is rate limiting in two-state protein folding of ACBP, Nat. Struct. Biol. 6, 594 (1999).
  30. E. I. Shakhnovich and A. M. Gutin, Engineering of stable and fast-folding sequences of model proteins., Proc. Natl. Acad. Sci. USA 90, 7195 (1993).
  31. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al., Training compute-optimal large language models, arXiv:2203.15556.
  32. L. Ghiringhelli, A. Zambon, and G. Tiana, Github repository (2026), https://github.com/guidotiana/PseudoLangevin.
  33. L. Ghiringhelli, A. Zambon, and G. Tiana, Supplementary material for: Sampling at intermediate temperatures is optimal for training large language models in protein structure prediction, UNIMI Dataverse, V1, 2026, https://doi.org/10.13130/RD_UNIMI/J1TOFK.
  34. M. Steinegger and J. Söding, MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets, Nat. Biotechnol. 35, 1026 (2017).
  35. J. Thickstun, The transformer model in equations, 2020, https://johnthickstun.com/docs/transformers.pdf.

Outline

Information

PRX Intelligence 1, 023003– Published 14 October, 2026
  • Received 31 March 2026
  • Accepted 21 September 2026
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation