Energetic electrons create distinct cathodoluminescence (CL) angular distributions upon interaction with dielectric and plasmonic nanostructures, providing valuable information on coherences in the excitation pathways. A counterintuitive prediction is that the CL signals arising from the excitation associated with different lateral regions of an extended electron wave are mutually in coherent and do not interfere, while the signals originating from different structures within the electromagnetic field of a narrow electron beam are mutually coherent. We present conclusive experimental evidence of these effects by examining the angular CL emission profile from defocused electron-beam excitation of a thin Si3N4 film, which is shown to follow an incoherent sum of CL excitations within the electron beam spot. In contrast, CL interferences are observed for separated plasmonic scatterers excited within the evanescent field of a single electron. Coherence may be recovered through correlations between emitted light and post-selected electron states, for which we propose a measurement geometry that erases which-path information of the electron trajectory.