- Featured in Physics
Scaling laws and representation learning in simple hierarchical languages: Transformers versus convolutional architectures
Phys. Rev. E 112, 065312 – Published 23 December, 2025
DOI: https://doi.org/10.1103/qtd6-nl8p
Abstract
How do neural language models acquire a language's structure when trained for next-token prediction? We address this question by deriving theoretical scaling laws for neural network performance on synthetic datasets generated by the random hierarchy model (RHM)—an ensemble of probabilistic context-free grammars designed to capture the hierarchical structure of natural language while remaining analytically tractable. Previously, we developed a theory of representation learning based on data correlations that explains how deep learning models capture the hierarchical structure of the data sequentially, one layer at a time. Here, we extend our theoretical framework to account for architectural differences. In particular, we predict and empirically validate that convolutional networks, whose structure aligns with that of the generative process through locality and weight sharing, enjoy a faster scaling of performance compared to transformer models, which rely on global self-attention mechanisms. This finding clarifies the architectural biases underlying neural scaling laws and highlights how representation learning is shaped by the interaction between model architecture and the statistical properties of data.
Physics Subject Headings (PhySH)
Collections
This article appears in the following collection:
Statistical Physics Meets Machine Learning - Machine Learning Meets Statistical Physics
The Editors of Physical Review E are pleased to present the Collection on Statistical Physics Meets Machine Learning - Machine Learning Meets Statistical Physics, highlighting research at the intersection of machine learning and statistical physics, on the occasion of the two Statistical Physics Meets Machine Learning and the two Machine Learning Meets Statistical Physics sessions at the 2025 Global Physics Summit. The Collection is being guest edited by David Schwab (CUNY, New York) and Yuhai Tu (IBM Watson Research Center Yorktown Heights, NY). Every article published in this collection underwent a rigorous peer review process, adhering to the same high standards applied to all papers. The Physical Review E editorial team managed the peer review and made all editorial decisions.
Viewpoint
Viewing Neural Networks Through a Statistical-Physics Lens
Statistical physics is shedding light on how network architecture and data structure shape the effectiveness of neural-network learning.
See more in Physics
