Quantum information concepts and quantum technologies have opened the prospect of probing the quantum nature of gravity in table-top experiments. Most proposals rely on detecting gravitationally mediated entanglement, which requires measurements on both interacting systems. Here we propose an alternative test based on witnessing quantum memory effects in the dynamics of a single system. We show how the quantum memory concept allows the non-classicality of the interaction between two systems to be certified through local measurements on only one of them. We then develop a novel quantum memory witness, making the approach applicable to gravitationally coupled harmonic oscillators. This witness requires no assumptions about the dynamics, such as Gaussianity. Our results show that the quantumness of gravitational dynamics can be certified with access to only one subsystem, opening the door to asymmetric implementations that can offer conceptual and experimental advantages to probing the quantum nature of the gravitational interaction.