Accurate reconstruction of peculiar velocities from galaxy positions is important for probing the motion and evolution of large scale structure. They are sensitive to the cosmological effects of gravity and dark sector matter, and complement other velocity inference methods such as the kinematic Sunyaev-Zel’dovich (kSZ) imprinted in the CMB. We show that machine learning methods improve velocity reconstruction by capturing nonlinear contributions. Specifically, we train both a gradient boosting decision tree (GBDT) and a Transformer using multi-scale features to predict the residual between the actual velocity and the estimate from linear theory for both the line-of-sight and transverse components. We evaluate our approach in both periodic box and, more realistic, lightcone settings using mock galaxy catalogs from the simulations tailored to DESI spectroscopic surveys of luminous red galaxies (LRGs) and emission line galaxies (ELGs). We also assess the impact of redshift uncertainties such as those in Rubin LSST photometry. Both models significantly outperform the linear theory with the Transformer achieving the best performance . They more accurately recover the density-weighted momentum power spectrum and maintain a higher cross-correlation with the true velocities across a wider range of spatial scales. Finally, we demonstrate two applications relevant to kSZ analyses: estimating cluster pairwise velocity correlations and stacked cluster density profiles. This machine learning framework for nonlinear velocity reconstruction opens up powerful new applications of survey data from DESI, Rubin LSST, Euclid and the Roman Space Telescope.