Scale-invariant feature extraction of neural network and renormalization group flow

Theoretical understanding of how a deep neural network (DNN) extracts features from input images is still unclear, but it is widely believed that the extraction is performed hierarchically through a process of coarse graining. It reminds us of the basic renormalization group (RG) concept in statistical physics. In order to explore possible relations between DNN and RG, we use the restricted Boltzmann machine (RBM) applied to an Ising model and construct a flow of model parameters (in particular, temperature) generated by the RBM. We show that the unsupervised RBM trained by spin configurations at various temperatures from $T=0$ to $T=6$ generates a flow along which the temperature approaches the critical value $T_c=2.27$. This behavior is the opposite of the typical RG flow of the Ising model. By analyzing various properties of the weight matrices of the trained RBM, we discuss why it flows towards $T_c$ and how the RBM learns to extract features of spin configurations.

Figure 1

Examples of spin configurations at temperatures $T=0,2,3,6$.

Figure 2

(a) Two-layer neural network of the RBM with a visible layer $\left\{v_i\right\}$ and a hidden layer $\left\{h_a\right\}$. These two layers are coupled but there are no intralayer couplings. (b) The RBM generates reconstructed configurations from $\left\{v_i\right\}$ to $\left\{{\tilde{v}}_i\right\}$ through the hidden configuration $\left\{h_a\right\}$.

Figure 3

Three-layer neural network for supervised learning with an input layer $\left\{z_i^{\left(1\right)}\right\}$, a hidden layer $\left\{z_a^{\left(2\right)}\right\}$, and an output layer $\left\{z_{\mu }^{\left(3\right)}\right\}$.

Figure 4

Training error and test error (up to 7500 epochs).

Figure 5

Probability distributions of measured temperatures for various sets of configurations generated at $T=0$, 2, 3, and 6. Temperature of the configurations can be distinguished by looking at the shapes of the distributions.

Figure 6

Temperature distributions after various numbers of iterations of type-V RBM, which is trained by the configurations at $T=0,0.25,...,6$. The original configurations are generated at $T=0$. After only several iterations, the temperature distribution peaks around $T_c$, and stabilizes there: $T_c$ is a stable fixed point of the flow.

Figure 7

Temperature distributions after various numbers of iterations of the same RBM as Fig. 6. The original configurations are generated at $T=6$. After $\sim 50$ iterations, the distribution stabilizes at $T_c$.

Figure 8

Temperature distributions after various numbers of iterations of the same RBM as Figs. 6 and 7. The original configurations are generated at $T=2.25$. The distribution is stable at around $T_c$.

Figure 9

Temperature distribution after various numbers of iterations of type-V RBM with 225 neurons in the hidden layer; i.e., $N_h>N_v$. The original configurations are generated at $T=0$. The distribution has a peak at $T=T_c$ after ten iterations, but then moves toward $T=\infty$.

Figure 10

Flow of temperature distributions starting from $T=0$ in type-H RBM that is trained by configurations at only $T=4,4.25,...,6$. The NN has (a) $N_h=64$ neurons and (b) $N_h=225$ neurons, respectively, in the hidden layer. Both flows move to higher temperature, and the speed is slower for the larger $N_h$.

Figure 11

Flow of temperature distributions starting from $T=6$ in type-L RBM. Type-L RBM is trained by configurations at only $T=0$. (a) $N_h=64$ and (b) $N_h=225$.

Figure 12

Elements of $W{W}^T$ when the hidden layer has (a) 16, (b) 100, and (c) 400 neurons.

Figure 13

Averaged values of the off-diagonal components of $W{W}^T$ (normalized by the diagonal components). The solid line (the most middle line) is the behavior of the type-V RBM that has learned all the temperatures $T=0,...,6$. Each of the other lines corresponds to the RBM that has learned configurations at a single temperature $T=0$, 2, 3, or 6, respectively.

Figure 14

Averaged values of $v^{\left(0\right)T}W{W}^T{v}^{\left(0\right)}$ over the 1000 input configurations at each temperature. Different lines correspond to type-V RBMs with different numbers of hidden neurons $N_h$. In this figure, the values at $T=6$ are subtracted for comparison between different RBMs.

Figure 15

Averaged values of $v^{\left(0\right)T}W{W}^T{v}^{\left(0\right)}$ over the 1000 configurations $\left\{v^{\left(0\right)}\right\}$ at each temperature. Quantities shown for the type-V RBM with (a) $N_h=64$ and (b) $N_h=225$.

Figure 16

Eigenvalues of $W{W}^T$ for type-V RBM (a smooth line) and type-L RBM (a steplike line). Both RBMs have 64 neurons in the hidden layer.

Figure 17

Eigenvalues of $W{W}^T$ for type-V RBM. The legend shows the number of hidden neurons, $N_h$.

Figure 18

Eigenvalues of $W{W}^T$ for type-L RBM. The legend shows the number of hidden neurons, $N_h$.