Generation of relativistic runaway electrons has historically not been considered a possible problem in stellarators, but this may not hold in reactor-scale stellarators despite the lack of an externally driven plasma current. The magnitude of the plasma current governs the exponential generation of runaways, and even if there is no externally driven plasma current, the bootstrap current could be considerable in reactor-relevant stellarators. In this paper, we present a pilot study on the generation of runaway electrons in stellarator temperature collapse scenarios. To reliably study runaway electrons in stellarators, we implemented a stellarator plasma model in the runaway electron simulation tool DREAM. The model is used to explore when runaway electrons can be generated with regard to combinations of initial plasma current, temperature decay time scale, and post-decay temperature. Special consideration is given to runaway generation through avalanche multiplication in stellarators, and how it compares to tokamaks. We find that significant runaway electron generation is possible also in stellarators, and demonstrate under which conditions it could be a concern. However, our findings support the conception that runaway electrons will be less of a concern in reactor-scale stellarators compared to tokamaks.