Short-sighted deep learning

A theory explaining how deep learning works is yet to be developed. Previous work suggests that deep learning performs a coarse graining, similar in spirit to the renormalization group (RG). This idea has been explored in the setting of a local (nearest-neighbor interactions) Ising spin lattice. We extend the discussion to the setting of a long-range spin lattice. Markov-chain Monte Carlo (MCMC) simulations determine both the critical temperature and scaling dimensions of the system. The model is used to train both a single restricted Boltzmann machine (RBM) network, as well as a stacked RBM network. Following earlier Ising model studies, the trained weights of a single-layer RBM network define a flow of lattice models. In contrast to results for nearest-neighbor Ising, the RBM flow for the long-ranged model does not converge to the correct values for the spin and energy scaling dimension. Further, correlation functions between visible and hidden nodes exhibit key differences between the stacked RBM and RG flows. The stacked RBM flow appears to move toward low temperatures, whereas the RG flow moves toward high temperature. This again differs from results obtained for nearest-neighbor Ising.

Figure 1

Block-spin averaging as introduced by Kadanoff. Groups of four red sites are averaged to obtain blue sites within their center.

Figure 2

Scaling dimensions of the spin lattice, with Hamiltonian as in Eq. (1) with $\mu =0$ and $\alpha =3$, determined using MCMC. ${\mathrm{\Delta }}_{\epsilon }$ is 1 and ${\mathrm{\Delta }}_s=0.53$ at $T=7.7$. These values are at the intercept of the vertical and horizontal red lines in plots (a) and (b). (a) ${\mathrm{\Delta }}_{\epsilon }$ versus temperature for a lattice of size of $10\times 10$. (b) ${\mathrm{\Delta }}_s$ versus temperature for a lattice size of $10\times 10$. (c) The magnetization-versus-temperature curve for various lattice sizes.

Figure 3

RBM flows starting from low-temperature ($T=0$) configurations. The RBM has $N_v=100$ visible nodes and $N_h=81$ hidden nodes. The training set is composed of 2000 configurations at each temperatures $T=0,0.5,\cdots ,14$, giving $30000$ configurations in total. (a) ${\mathrm{\Delta }}_s$ versus flow length. (b) ${\mathrm{\Delta }}_{\epsilon }$ versus flow length. (c) The average probability of temperature versus temperature of flows of various lengths. The flow tends to a temperature of 6. The red vertical line indicates the critical temperature of 7.7.

Figure 4

RBM flows starting from configurations near the critical temperature ($T_c\approx T=7.7$). The RBM has $N_v=100$ visible nodes and $N_h=81$ hidden nodes. Training uses 2000 configurations at temperatures $T=0,0.5,\cdots ,14$, giving $30000$ configurations in total. (a) ${\mathrm{\Delta }}_s$ versus flow length. (b) ${\mathrm{\Delta }}_{\epsilon }$ versus flow length. (c) The average probability of temperature versus the temperature of flows of various lengths. The red vertical line indicates the critical temperature of 7.7.

Figure 5

RBM flows starting from configurations at a high temperature ($T=14$). The RBM network has $N_v=10\times 10=100$ visible nodes and $N_h=9\times 9=81$ hidden nodes. The training set is made up of configurations at temperatures of $T=0,0.5,\cdots ,14$ with 2000 configurations at each temperature (in total $30000$ configurations). (a) ${\mathrm{\Delta }}_s$ versus flow length. (b) ${\mathrm{\Delta }}_{\epsilon }$ versus flow length. (c) The average probability of temperature versus temperature of flows of various lengths. The red vertical line indicates the critical temperature of 7.7.

Figure 6

Plots showing the average temperature of flows of different lengths for different values of $\alpha$. (a) Results for $\alpha =4$, (b) results for $\alpha =5$, and (c) results for $\alpha =6$.

Figure 7

Plots showing the percentage of samples that are near the critical temperature for different flow lengths and different values of $\alpha$. (a) Results for $\alpha =3$, (b) results for $\alpha =4$, (c) results for $\alpha =5$, and (d) results for $\alpha =6$.

Figure 8

RG $〈vh〉$ correlation plot. Plots (a), (b), (c), and (d) show typical $〈v^{\left(1\right)}{h}^{\left(1\right)}〉$ plots where $v^{\left(1\right)}$ is an input spin and $h^{\left(1\right)}$ is a block spin produced by one step of RG. Plots (e), (f), (g), and (h) show typical $〈v^{\left(1\right)}{h}^{\left(2\right)}〉$ plots where $v^{\left(1\right)}$ is an input spin and $h^{\left(2\right)}$ is a block spin produced by two steps of RG. Plots (a), (b), (c), and (d) show typical $〈v^{\left(1\right)}{h}^{\left(3\right)}〉$ plots where $v^{\left(1\right)}$ is an input spin and $h^{\left(3\right)}$ is a block spin produced by three steps of RG. In all 12 plots a single block spin's correlator with the complete collection of $64\times 64$ input spins is plotted.

Figure 9

RBM $〈vh〉$ correlation plot. Plots (a), (b), (c), and (d) show a sample of the $〈v^{\left(1\right)}{h}^{\left(1\right)}〉$ plots where $v^{\left(1\right)}$ is an input spin and $h^{\left(1\right)}$ is a hidden spin produced by the first RBM in the stacked network. Plots (e), (f), (g), and (h) show a sample of the $〈v^{\left(1\right)}{h}^{\left(2\right)}〉$ plots where $v^{\left(1\right)}$ is an input spin and $h^{\left(2\right)}$ is a hidden spin produced by the second RBM in the stacked network. Plots (a), (b), (c), and (d) show a sample of the $〈v^{\left(1\right)}{h}^{\left(3\right)}〉$ plots where $v^{\left(1\right)}$ is an input spin and $h^{\left(3\right)}$ is a hidden spin produced by the third RBM in the stacked network. In all 12 plots a single hidden node's correlator with the complete collection of $64\times 64$ input spins is plotted.

Figure 10

Average probability of temperature versus temperature of a flow for a stacked RBM network trained on lattices at $T=7.7$ and an RG flow of three steps. The RG flow and RBM flows are generated by applying both methods to the training set of $30000$ lattices at $T=7.7$ consisting of $64\times 64$ spins. (a) Results for the stacked RBM flow and (b) results for the RG flow.

Figure 11

The lines labeled RBM show the log of average values of off-diagonal components, $x_{i,i}$ versus the distance between spins $v_i$ and $v_j$ given by $|{\mathbf{i}}_{\mathbf{s}}-{\mathbf{j}}_{\mathbf{s}}|$. The weight matrices are from RBMs with $N_v=10\times 10$ and $N_h=9\times 9$ trained on long-range spin-chain data at $T=14$ and the specified values of $\alpha$. The line labeled $\alpha =2.3$ shows the log of a power-law falloff of $1/|{\mathbf{i}}_{\mathbf{s}}-{\mathbf{j}}_{\mathbf{s}}{|}^{\alpha }$ with $\alpha =2.3$.

Figure 12

Histograms showing the visible biases for RBMs trained on high temperature ($T=14$) long-range spin-chain data with $N_v=10\times 10$ and $N_h=9\times 9$. Each plot shows results for a different value of $\alpha$. (a) Results for $\alpha =3$, (b) results for $\alpha =4$, (c) results for $\alpha =5$, and (d) results for $\alpha =6$.

Figure 13

Energy histograms for 2000 pairs of visible and hidden vectors for RBMs trained on long-range spin-chain data at $T=14$ for various values of $\alpha$. The RBM networks have $N_v=10\times 10$ and $N_h=9\times 9$. Each plot shows results for a different value of $\alpha$. (a) Results for $\alpha =3$, (b) results for $\alpha =4$, (c) results for $\alpha =5$, and (d) results for $\alpha =6$.