Neural networks are complex functions of both their inputs and parameters. Much prior work in deep learning theory analyzes the distribution of network outputs at a fixed set of inputs (e.g., a training dataset) over random initializations of the network parameters. The purpose of this article is to consider the opposite situation: We view a randomly initialized multilayer perceptron (MLP) as a Hamiltonian over its inputs. For typical realizations of the network parameters, we study the properties of the energy landscape induced by this Hamiltonian, focusing on the structure of near-global minimum in the limit of infinite width. Specifically, we use the replica trick to perform an exact analytic calculation giving the entropy (log volume of space) at a given energy. We further derive saddle-point equations that describe the overlaps between inputs sampled independent and identically distributed from the Gibbs distribution induced by the random MLP. For linear activations we solve these saddle-point equations exactly. But we also solve them numerically for a variety of depths and activation functions, including $tanh,sin,\text{ReLU}$, and shaped nonlinearities. We find even at infinite width a rich range of behaviors. For some nonlinearities, such as $sin$, for instance, we find that the landscapes of random MLPs exhibit full replica symmetry breaking, while shallow $tanh$ and ReLU networks or deep shaped MLPs are instead replica symmetric.

Close

PrevNext

View figure in article

• Artificial neural networks
• Machine learning categories