In this work we realize the $3+1$d non-invertible $\Z_N$ chiral symmetry generator as an operator in a many body lattice Hilbert space. A crucial ingredient in our construction is the use of infinite dimensional $U(1)$ rotor site Hilbert spaces. Specifically, our Hilbert space is that of a $U(1)$ lattice gauge theory coupled to a charge $1$ scalar in the Villain formulation, which allows for direct access to monopoles and for a simple definition of a magnetic $\Z_N$ one-form symmetry $Z_m^{(1)}$, at the lattice Hamiltonian level. We construct the generator of the $\Z_N$ chiral symmetry as a locality-preserving unitary operator not in the entire Hilbert space, but only in the subspace of $Z_m^{(1)}$-invariant states. Using a lattice-level duality based on gauging $Z_m^{(1)}$, we find a dual description of this subspace, as the subspace of a charge $\frac{1}{N}$ gauge theory invariant under an electric one-form symmetry $Z_e^{(1)}$. We show that in this dual formulation, the chiral symmetry generator extends unitarily to the entire Hilbert space, but has a mixed anomaly with the $Z_e^{(1)}$ symmetry.