Stochastic gradient descent (SGD) is a fundamental tool in stochastic optimization that is widely used in machine learning. Here, we study SGD through the lens of particle dynamics, introducing a framework that links its behavior to well-established models in nonequilibrium statistical physics. We introduce a minimal model in which SGD is applied to spherical particles in physical space, minimizing the system's energy by stochastically reducing particle overlaps. This process exhibits an absorbing phase transition, prompting us to adopt the framework of biased random organization (BRO), a nonequilibrium absorbing state model, to describe SGD's dynamical behavior. We show that BRO dynamics can be approximated by those of particles with linear repulsive interactions under multiplicative anisotropic noise. Thus, in the limit of small kick sizes (learning rates), we demonstrate that BRO and SGD of linear repulsive particles become equivalent, converging to the same critical packing fraction ${\varphi }_c\approx 0.64$, despite their distinct noise mechanisms. This equivalence is further supported by the observation that both models exhibit behavior consistent with the Manna universality class near criticality. Above the transition, SGD tends to favor flatter regions of the energy landscape, analogous to the solutions it finds during neural network training that are associated with improved generalization.

Close

PrevNext

View figure in article

• Fluctuations & noise
• Nonequilibrium statistical mechanics