Storage ring-based steady-state microbunching (SSMB) is a promising approach for generating high-average-power coherent radiation, while the instabilities driven by coherent undulator radiation in the laser modulator (LM) is important for the ring performance. In this paper we investigate the longitudinal single-bunch multi-turn LM instability using cavity mode decomposition techniques. The evolution of the wakefield in the longitudinal beam dynamics equations are derived, and the instability growth rates are analyzed. Numerical simulations show excellent agreement with the theoretical model, validating the mode decomposition approach. These findings provide critical insights into the design and operation of SSMB storage rings, suggesting effective mitigation strategies to suppress the instability and enhance the overall performance.