Local local oscillator (LLO) continuous-variable quantum key distribution (CVQKD) systems have gained prominence due to their security advantages with LO generated locally. However, the nonsynchronization of two lasers leads to phase drift, a detrimental phenomenon that restricts excess noise suppression. Prevailing solutions, such as pilot-multiplexing schemes, offer better recovery results than pilot-sequential schemes, but at the expense of increased system complexity and demanding technical requirements. In this work, we introduce an innovative carrier-recovery strategy employing the long-short-term memory (LSTM) neural network, aiming to optimize the recovery performance in the simple self-referenced CVQKD systems with the pilot-sequential scheme. Through temporal modeling, the LSTM network proficiently predicts and compensates for the rapid phase drifts. Experimental validations conducted in both fiber and free-space channels underscore the effectiveness of our method for carrier recovery. In practical CVQKD experiments, our LSTM-based approach yields a near 50% reduction in excess noise, leading to a significant increase in secret key rates when compared with the traditional pilot-sequential method. The simplicity of hardware and operation positions our scheme as a superior alternative for current mainstream pilot-multiplexing CVQKD systems, particularly appealing for on-chip implementation and satellite-to-ground quantum communication applications.