Cavity materials engineering, aiming to manipulate material properties by coupling to vacuum fluctuations inside a cavity, is a rapidly advancing field. Despite significant progress, most studies to date have focused on specific materials and cavity configurations. Here, through a comprehensive group-theoretical analysis, we establish general symmetry rules for cavity materials engineering with linearly polarized cavity photon modes. By analyzing the symmetry of the Pauli-Fierz Hamiltonian within the long-wavelength approximation, we provide a classification of the symmetry-breaking patterns induced by cavity modes for all crystallographic point groups. The power of this framework is then demonstrated by quantum-electrodynamical density functional theory calculations. In particular, we explain the distinct cavity-induced lifting of band degeneracies in cubic BaTiO${}_3$ for different cavity mode configurations, and the cavity-modified infrared and Raman spectra of monolayer MoS${}_2$ due to symmetry breaking. Our results highlight the central role of symmetry in cavity materials engineering and provide general guidelines for future studies in this field.