We investigate the effect of remote band mixing on the stability of fractional Chern insulators in a family of models that approximate continuum descriptions of moiré materials. Our results suggest that the experimentally observed asymmetry between filling fractions $\nu =1/3$ and $\nu =2/3$ in twisted MoTe${}_2$ originates from a competition between a fractional Chern insulator, an electron Wigner crystal, and a hole Wigner crystal. In the absence of band mixing, the leading instability at $\nu =1/3$ is the electron crystal, whereas at $\nu =2/3$ the main competing phase is the hole crystal. Remote band mixing substantially lowers the energy of the electron crystal but has only a weak effect on the hole crystal. Consequently, it brings competing crystalline states much closer in energy to the fractional Chern insulator at $\nu =1/3$ than at $\nu =2/3$, leading to a stronger suppression of the many-body gap. This mechanism also provides an explanation for the emergence of re-entrant integer quantum anomalous Hall states in moiré MoTe${}_2$ for fillings $\nu >1/2$.