A single-cycle terahertz (THz) pulse applied to single-crystal holmium iron garnet Ho${}_3$Fe${}_5$O${}_{12}$ strongly perturbs the magneto-optical Faraday effect – an effect conventionally employed for optical readout of the magnetic state. This ultrafast perturbation sets in when cooling below the magnetization compensation temperature, where the unit cell distorts and the spins of Ho${}^{3+}$ form a noncollinear "double-umbrella" structure. The experimental dependencies reveal that the findings can be described as the ultrafast THz-electric-field-induced magneto-optical (TEMO) Faraday effect, caused by the resonant excitation of crystal-field-split $f$-$f$ electronic transitions of Ho${}^{3+}$ in a noncentrosymmetric environment of oxygen ions. We demonstrate that the effect is not limited to Ho${}_3$Fe${}_5$O${}_{12}$, but is also present in other rare-earth iron garnets, such as Tb${}_3$Fe${}_5$O${}_{12}$. The effect can facilitate ultrafast optical readout of magnetic bits at a sub-$100$~nm spatial scale.