We report a family of two-dimensional (2D) rectangular $\gamma$-X (X=N, P, As, Sb, Bi) materials that simultaneously exhibit topologically nontrivial properties and ultra-large, strain-sensitive negative Poisson’s ratios (NPRs). According to first-principles calculations, the $\gamma$-X single layers (SLs) exhibit excellent mechanical, dynamic, and thermodynamic stability, with strongly anisotropic electronic structures and mechanical properties. At equilibrium lattice, $\gamma$-Bi SL is a 2D topological insulator (TI), which has a PBE band gap of $0.06$ eV and a HSE06 energy gap of $0.13$ eV. Though $\gamma$-N, P, As, and Sb SLs are trivial insulators, biaxial tensile strain can trigger a topological phase transition, transforming them into 2D TIs. Mechanically, $\gamma$-P, As, Sb, and Bi SLs exhibit NPRs ranging from $-0.002$ to $-0.066$, and all the SLs, including $\gamma$-N SL, show strongly strain-dependent in-plane and out-of-plane NPRs. Remarkably, the uniaxial compression along the armchair direction induces a giant out-of-plane NPR of $-2.864$ in $\gamma$-P SL, far exceeding the NPR values in previously reported auxetic materials. Moreover, these SLs can withstand ultimate tensile strains along the zigzag direction ranging from $2\%$ to $52\%$. The van der Waals layered 3D counterparts of $\gamma$-X also exhibit strong anisotropy and NPRs, with maximum magnitudes between $-0.052$ and $-1.290$.