We compute the amplitude for the scattering of a gravitational wave off of a spinless point particle at fourth order in Newton’s constant, using the worldline quantum field theory formalism. A decomposition of our master integrals incorporating Murua coefficients allows us to entirely bypass the cut subtraction needed to convert the scattering amplitude into the Magnusian, the latter mapping directly onto the scattering phase shift in partial wave space. Elliptic functions appear in momentum space for the first time for this process at this order. To compare to the phase shift from black hole perturbation theory, we present an analytical method for performing the partial wave projection of the new elliptic sector to any partial wave order $\ell$, and verify the equivalence of the Magnusian and the phase shift. This demonstrates that point-particle worldline quantum field theory accurately describes Schwarzschild black holes up to $\cO(G^4)$.