The interaction of free-electron wave packets with electromagnetic fields provides a powerful route toward coherent electron control, enabling the generation of energy combs, momentum-state superpositions, and aberration-engineered electron beams. Existing theoretical descriptions, however, often rely on eikonal or no-recoil approximations. Here, we present a mesh-free numerical framework that directly solves the time-dependent single-particle Schrödinger equation for arbitrary electromagnetic potentials. Comparison with a benchmark mesh-based Schrödinger solver reveals excellent quantitative agreement. By eliminating the need for spatial meshing, our method offers an efficient and scalable route for simulating electron wave packet dynamics in complex time-dependent and static electromagnetic environments, while the simulation time is significantly improved by up to 800 times faster. These capabilities establish a versatile computational tool for quantum electron optics and free-electron-light interactions beyond eikonal approximations.