Stochastic Thermodynamics of Learning

Virtually every organism gathers information about its noisy environment and builds models from those data, mostly using neural networks. Here, we use stochastic thermodynamics to analyze the learning of a classification rule by a neural network. We show that the information acquired by the network is bounded by the thermodynamic cost of learning and introduce a learning efficiency $\eta \le 1$. We discuss the conditions for optimal learning and analyze Hebbian learning in the thermodynamic limit.

Figure 1

Model of a single neuron. Given a set of inputs ${\mathit{\xi }}^{\mu }\in \{\pm 1{\}}^N$ and their true labels ${\sigma }_T^{\mu }=\pm 1$ (left), the neuron learns the mappings ${\mathit{\xi }}^{\mu }\to {\sigma }_T^{\mu }$ by adjusting its weights $\mathit{\omega }\in {\mathbb{R}}^N$. It processes an input by computing the activation ${\mathcal{A}}^{\mu }=\mathit{\omega }·{\mathit{\xi }}^{\mu }/\sqrt{N}$, which determines the transition rates of a two-state random process ${\sigma }^{\mu }=\pm 1$ indicating the label predicted by the neuron for each sample, shown here for $\mu =1$.

Figure 2

Learning efficiency of a neuron with a single weight. We plot the efficiency $\eta$ (11) for a neuron with a single weight learning a single sample as a function of the learning rate $\nu$ and learning duration $\tau$ in the limit $t\to \infty$.

Figure 3

Hebbian learning in the thermodynamic limit. We plot the mutual information between the true and predicted label of a randomly chosen sample (21) in the limit $t\to \infty$ with $N,P\to \infty$ as a function of $\alpha \equiv P/N$, computing $p_C^{\mu }$ from Eq. (20) (solid lines) and by Monte Carlo integration of $p({\mathit{\sigma }}_T,\mathit{\omega },\mathit{\sigma })$ (crosses; the error bars indicate 1 standard deviation). The inset shows the learning efficiency (22) in the limits $\tau \to 0$ (solid) and $\tau \to \infty$ (dashed). In both plots, $\nu$ increases from bottom to top.