We present experimental evidence of a thermally-driven amplitude-frequency nonlinearity in a thin-film quartz phononic crystal resonator at millikelvin temperatures. The nonlinear response arises from the coupling of the mechanical mode to an ensemble of microscopic two-level system defects (TLS) driven out of equilibrium by the microwave readout signal. In contrast to the conventional Duffing oscillator, the observed nonlinearity exhibits a mixed reactive-dissipative character. Notably, the reactive effect can manifest as either a softening or hardening of the mechanical resonance, depending on the ratio of thermal to phonon energy. By combining the standard TLS theory with a thermal conductance model, the measured power-dependent response is quantitatively reproduced and readout-enhanced relaxation damping from off-resonant TLSs is identified as the primary mechanism limiting mechanical coherence. Within this framework, we delineate the conditions under which similar systems will realize this nonlinearity.