The mechanical response of soft materials depends on their deformation history, and identifying that memory from experiments is essential for predicting how these materials behave within the linear viscoelastic regime. Chirp rheometry has recently emerged as a way to accelerate this characterization, compressing hours of conventional measurement into seconds and yielding thousands of stress–strain pairs per experiment. That density is then largely discarded: the standard pipeline reduces the record to a handful of frequency-domain estimates before any linear viscoelastic model is fitted. We introduce a physics-informed Gaussian process framework that infers the linear viscoelastic memory kernel directly from the raw time-domain record of a single chirp, selecting among a prescribed library of candidate memory kernels and parametrizing the selected one without any intermediate spectral-estimation step. Because the framework infers the memory kernel rather than the specific waveform used during training, it predicts the response to deformation histories it never saw, without retraining. From a two-second chirp, the inferred memory places the terminal crossover of a wormlike micellar solution below the excited band, in agreement with an independent, longer chirp, thereby extending the recovered relaxation spectrum beyond the frequencies used for training. The method also resolves material evolution within a single chirp directly in the time domain.