We show how to define a quantized many-body charge polarization for topological phases of matter, even in the presence of nonzero Chern number and magnetic field. For invertible topological states, is a , , , or topological invariant in the presence of (, 3, 4, or 6)-fold rotational symmetry, lattice (magnetic) translational symmetry, and charge conservation. manifests in the bulk of the system as (i) a fractional quantized contribution of to the charge bound to lattice disclinations and dislocations with Burgers vector , (ii) a linear momentum for magnetic flux, and (iii) an oscillatory system size dependent contribution to the effective 1D polarization on a cylinder. We study in lattice models of spinless free fermions in a magnetic field. We derive predictions from topological field theory, which we match to numerical calculations for the effects (i)–(iii), demonstrating that these can be used to extract from microscopic models in an intrinsically many-body way. We show how, given a high symmetry point o, there is a topological invariant, the discrete shift , such that specifies the dependence of on o. We derive colored Hofstadter butterflies, corresponding to the quantized value of , which further refine the colored butterflies from the Chern number and discrete shift.