The detection of Cosmic Microwave Background primordial $B$-mode polarization would constitute a ``smoking gun" signal of primordial gravitational waves. However, this measurement requires accurate removal of polarized Galactic foregrounds to avoid systematic biases when estimating the tensor-to-scalar ratio. Methods based on Machine Learning (ML) techniques, such as Convolutional Neural Networks (CNNs), have recently been proposed as alternative foreground cleaning techniques, but their applicability to real data relies on their ability to generalize beyond the models assumed during training. In this work, we focus on a variety of foreground models (FMs) used for training and conduct a systematic study of the generalization properties of a CNN-based method. We train two CNN architectures on simulations generated from different Galactic FMs, and test their performance on models not used during the training. By characterizing the pixel-to-pixel statistical properties of the FMs using variance, skewness, and Shannon entropy, we define a marginal statistical complexity hierarchy among them. The morphology diversity of the models is also incorporated in the hierarchy established. We show that training on the more complex FMs reduces bias and improves precision when testing on unseen template-based FMs, whereas training on the simplest model could introduce systematic errors, reaching a residual equivalent to $r>0.1$ on both models used only for testing. We further test this picture by constructing controlled modifications of the simplest FM with progressively broader marginal distributions. While their marginal statistics grow beyond those of the most morphologically diverse model, their generalization improves only up to a point and none of them matches the robustness across test models of the more complex model, showing that marginal variability alone is not sufficient for generalization. These results evidence that a lack of generalization is a relevant source of systematic uncertainty, and emphasize the importance of understanding the impact of the models assumed during training in ML-based methods before applying them to real data.