Temporal characterization of complex modulated ultrafast electron beams (UEBs) is crucial for optimizing X-ray free-electron lasers (XFELs) and ultrafast science applications. Although the coherent radiation spectra encode the temporal information of the modulated UEBs, there is a lack of a reliable temporal reconstruction algorithm that can simultaneously retrieve the femtosecond-scale profile and attosecond-scale substructures. In this paper, adopting a sampling-based perspective for analyzing the coherent radiation spectra of modulated UEBs, we propose a novel cross-scale temporal reconstruction algorithm that separately reconstructs the high- and low-frequency components. Tests on the UEBs from a complex multi-Gaussian model and bunch-enhanced coherent harmonic generation demonstrate that the proposed algorithm achieves performance superior to the conventional retrieval algorithm. This method paves the way for the application of complex modulated electron beams in XFELs and ultrafast science.