We use whispering-gallery-mode analysis to characterise the microwave dielectric properties of single-crystal TeO2 at cryogenic temperatures and compare its loss performance with other low-loss dielectric materials. Finite-element modelling is combined with measurements at room temperature, 4 K, and 20 mK to develop accurate cryogenic simulations and extract the anisotropic dielectric permittivities, giving ε∥ = 25.75±0.08 and ε⊥ = 20.90±0.07. Loss measurements reveal quality factors as high as 9 × 10^6 and minimum loss tangents approaching 3 × 10^−8, placing TeO2 among promising low-loss dielectrics for cryogenic microwave applications. Electron-spin-resonance spectroscopy further indicates a clean spin environment, while identifying distinct spin systems consistent with the known properties of the crystal.