We develop a diagrammatic approach analyzing Gibbs measures (i.e., Bayesian posteriors) in deep shaped multilayer perceptrons at arbitrary temperature. This gives the first (perturbatively) solvable model of learning with nonlinear neural networks where the input dimension $N_0$, depth $L$, width $N$, and number of samples $P$ can all be large without severe assumptions on either initialization or training data statistics. The limits $N_0,N,L,P\to \infty$ do not commute, resulting in a rich phase diagram of learning regimes that we study to first order in $1/N$. We find in particular that the ratio $LP/N$ defines a critical depth necessary for feature learning: if $LP/N\to 0$, then Bayesian posteriors coincide with those of a kernel method. Regimes where $LP/N\to \lambda >0$, in contrast, correspond to learning with data-dependent deformations of these kernels, and we provide explicit formulas for the resulting features to first order in $1/N$.

Close

PrevNext

View figure in article

• Artificial neural networks
• Deep learning
• Langevin algorithm