Emergence of Compositional Representations in Restricted Boltzmann Machines

Extracting automatically the complex set of features composing real high-dimensional data is crucial for achieving high performance in machine-learning tasks. Restricted Boltzmann machines (RBM) are empirically known to be efficient for this purpose, and to be able to generate distributed and graded representations of the data. We characterize the structural conditions (sparsity of the weights, low effective temperature, nonlinearities in the activation functions of hidden units, and adaptation of fields maintaining the activity in the visible layer) allowing RBM to operate in such a compositional phase. Evidence is provided by the replica analysis of an adequate statistical ensemble of random RBMs and by RBM trained on the handwritten digits data set MNIST.

Figure 1

(a) Architecture of RBM. Visible ($v_i$, $i=1,\dots ,N$) and hidden ($h_{\mu }$, $\mu =1,..,M$) units are connected through weights ($w_{i\mu }$). (b) Activation functions $\mathrm{\Phi }$ of Bernoulli, linear, and rectified linear units. The corresponding potentials are ${\mathcal{U}}^{\text{Lin}}(h)=(h^2/2)$; ${\mathcal{U}}^{\text{Ber}}(h)=h{\theta }_B$ if $h=0$ or 1, and $+\infty$ otherwise; ${\mathcal{U}}^{\text{ReLU}}(h)=(h^2/2)+h\theta$ for $h\ge 0$, $+\infty$ for $h<0$. (c) The three regimes of operation; see the text. Black, grey, and white hidden units symbolize, respectively, strong, weak, and null activations.

Figure 2

Training of RBM on MNIST, with $N=28\times 28$ visible units and $M=400$ hidden ReLU. (a) Set of weights ${\mathbf{w}}_{\mu }$ attached to four hidden units $\mu$. (b) Averages of $\mathbf{v}$ conditioned to five hidden-unit configurations $\mathbf{h}$ sampled from the RBM at equilibrium. Black and white pixels correspond respectively to averages equal to 0 and 1; few intermediary values, indicated by grey levels, can be seen on the edges of digits. (c) Distributions of $\widehat{L}$ (left) and $\widehat{S}$ (right) in hidden-unit configurations at equilibrium. (d) Evolution of the weight sparsity $\widehat{p}$ (red) and the squared weight value $W_2$ (blue); the training time is measured in epochs (number of passes over the data set), and represented on a square-root scale.

Figure 3

Compositional regime in R-RBM. (a) Critical lines in the $\theta$, $\alpha$ plane below which $L$ hidden units may be strongly activated, calculated from the optimization of $E_{\mathrm{GS}}$ (2). Parameters are $p_i=p=0.1$, $g=-0.02$. (b) Comparison of theoretical (red crosses) and numerical simulations ($N=10^4$ visible units, colored points) for the rescaled magnetizations $\tilde{m}=m/(p/2)$ as a function of the number $L$ of strongly activated hidden units in R-RBM. 7,500 zero temperature MCMC, each initialized with a visible configuration strongly overlapping with $L=1,2,\dots$ features, were launched; the color code indicates the probability that the same $L$ hidden units are magnetized after convergence (see the bottom scale), and the corresponding average magnetization $\tilde{m}$.

Figure 4

(a) Behavior of the GS energy $e_{\ell }$ vs $\ell =L\times p$ in the $p\to 0$ limit. Parameters: $\tilde{\theta }=1.5$, $\alpha =0.5$. [(b)–(d)] Quantitative predictions in the compositional regime of R-RBM compared to RBMs inferred on MNIST. Each point represents a ReLU RBM trained with various regularizations, yielding different weight sparsities $p$. Solid lines depict predictions found by optimizing $e_{\ell }$ (2), and dashed line expected fluctuations at finite size ($N$) and temperature. (b) Average number $L$ of active hidden units vs $p$. (c) Average magnetization $\tilde{m}$ vs $\ell =L\times p$. (d) Distance (per pixel) between the pairs of visible configurations after convergence of zero temperature MCMC vs relative distances in the hidden-unit activation patterns. MCMC are initialized with all pairs of digits in MNIST; final visible configurations differ from MNIST digits by about seven pixels, both on the training and test sets; see Supplemental Material [24], Fig. 1. In all four panels predictions for the homogeneous ($\tilde{g}=-0.21$) and heterogeneous ($\tilde{g}=-0.1725$; see Supplemental Material [24], Sec. III E) cases are shown in, respectively, cyan and magenta.