Spin relaxation results in the spin decoherence and a finite spin lifetime, which are detrimental to spintronic devices. To achieve a long spin lifetime desirable for spintronic devices, elucidating the spin relaxation mechanism and factors influencing the spin lifetime is of vital importance. Here, we investigate the spin relaxation in $X$-wave magnets ($X=p,d,f,g,i$) with Rashba spin–orbit coupling within the framework of D’yakonov-Perel’ mechanism. We calculate the tensor of the spin relaxation rate for an arbitrary N'eel vector direction in the regime of weak spin-orbit field. It is found that the spin relaxation rate matrices are anisotropic and can be dictated by the magnetic point group symmetry. As an illustration, we study the spin relaxation for the N'eel vector along the $[001]$ direction. For the $p$-wave magnet, we derive the analytical expressions for the temporal evolution of spins. Moreover, we show that the spin relaxation is highly controllable by the N'eel vector and the spin relaxation rate is proportional to the momentum relaxation time, Rashba and spin-split strengths. Our results shine more light on the fundamental understanding of the spin relaxation mechanism in unconventional $X$-wave magnets.