Hybrid quantum devices using surface acoustic waves show promise as key elements of quantum information processors. We report measurements of integrated flip-chip devices consisting of semiconductor quantum dots and surface acoustic wave resonators in lithium niobate. We observed that the pyroelectric effect in lithium niobate inhibited the operation of quantum dots in the integrated devices. GaAs/AlGaAs devices suffered from unintentional carrier depletion, and Si/SiGe devices suffered from electrostatic discharge. Our results highlight the importance of mitigating pyroelectric effects in semiconductor-lithium niobate hybrid devices for continued progress in quantum interconnects and transducers.