Zonal modes are central to magnetic confinement because their radial shears regulate turbulence and transport. While the generation of these flows is well understood, the mechanisms limiting their persistence in collisionless regimes remain unresolved. In this paper, we demonstrate that nonlinear spectral back-transfer of free energy from zonal modes to turbulence is a major nonlinear process that limits the temporal coherency of the shearing field. Back-transfer events induce stochastic phase and amplitude scattering of zonal shear that limits its auto-coherence time. Using local gyrokinetic GENE simulations for collisionless electrostatic ion temperature gradient (ITG) driven turbulence with adiabatic electrons for parameters representative of core conditions, we show that back-transfer is highly intermittent and occurs in bursts that co-exist with the zonal flow generation process. The probability distribution of the zonal free energy transfer is non-Gaussian, with positive triangularity (PT) exhibiting substantially higher kurtosis than negative triangularity (NT), reflecting the markedly more intermittent and heavy-tailed character of back-transfer bursts in PT. We find that NT plasmas exhibit significantly reduced back-transfer compared to PT. This suppression increases the shear auto-coherence time {E} and the shearing Kubo number K{u}, leading to more resilient and effective turbulence regulation despite lower absolute zonal kinetic energy. These results identify back-transfer as a key nonlinear damping mechanism and suggest that it must be explicitly treated in reduced models of drift-wave zonal-flow turbulence.