We identify a previously unrecognized spin-lattice coupling allowed by broken inversion symmetry that can be viewed as a lattice analogue of electronic Rashba spin-orbit coupling. In the low-frequency regime, where magnons are integrated out, its interplay with the conventional magnetoelastic coupling generates a kineo-elastic term that couples strain and lattice velocity, giving rise to nonreciprocal phonon velocity. We further analyze the full magnon-phonon spectrum and find that the competition between interfacial spin-lattice and magnetoelastic couplings leads to directional hybridization and absorption, yielding asymmetric phonon propagation lengths for opposite directions. Our results indicate that such interfacial spin-lattice coupling can provide an efficient route to nonreciprocal phonon propagation in magnetic heterostructures with strong Rashba spin-orbit coupling.