We show that in rotationally invariant spin-$3/2$ systems the orbital Hall effect originates from interband matrix elements of the orbital magnetic moment. The resulting Hall response is governed by the quantum metric, rather than by the Berry curvature, revealing a purely geometric transport mechanism. Conventional intraband contributions associated with the orbital magnetic moment and Berry curvature are shown to vanish identically in the doubly degenerate two-band structure of the Luttinger model. These results identify the quantum metric as the key geometric quantity controlling orbital Hall transport in degenerate spin-$3/2$ systems.