Cosmological inference using a population of binary black-hole mergers, combined with a galaxy catalog, presents an exciting opportunity for precision cosmology with the possibility of resolving the Hubble tension. However, the accuracy of these measurements heavily relies on the quality of the model used to infer the binary parameters, including the model of the gravitational-wave signal. We use state-of-the-art quasi-circular, multipolar, spin-precessing waveform models, including mode asymmetries, to explore the impact of inaccurate modeling in measuring the Hubble-Lemaître constant for the upcoming and future ground-based gravitational-wave observatories. We diagnose the presence of inaccuracies within a hierarchical population analysis framework, without a priori knowing the true value of the parameter, by assessing the consistency of the distribution of individual posteriors in relation to their measurement errors. Our findings indicate that even a small high-mass, spin-precessing subpopulation—comprising as little as 2% of the population generating the events observed by the LIGO-Virgo-KAGRA Collaboration so far—can result in an unreliable measurement of the Hubble-Lemaître constant in the upcoming observing runs of these detectors, with even more pronounced effects expected in future facilities on the ground. Crucially, we find systematic biases across a wide range of waveform models, rather than for only a specific signal-template pair.