In invasive breast carcinomas, active cancer cell clusters of varying sizes and shapes are embedded in a nematic extracellular matrix (ECM) of aligned fibres. Although the tissue patterns are regularly used in diagnosis, the physical mechanisms by which these mesoscale structures emerge on long time scales remain unexplored, partly because in-vitro or in-vivo experiments over the timescale of cancer progression (~months-years) are not feasible. To overcome this fundamental problem in studying tumour dynamics, we model the cancer clusters as active nematic aggregates moving through the ECM, which acts as a confining, passive nematic phase. The approach is verified by comparison to histological images showing hundreds of thousands of cancer clusters and their micro-environments from over 2,000 breast cancer patients. We predict that collective cancer cell motion leads to cancer cluster splitting and coalescence, which is manifested in specific scaling behaviours for cancer cluster shape and size distributions. Active stresses from cancer cluster movement give rise to topological defects in the ECM and tend to align the passive fibres parallel to the cluster boundary. We further show that cluster shapes and defect distributions as measures of increased activity are associated with poorer survival probabilities for breast cancer patients. Our work provides a novel way to study human breast cancer by linking long-timescale, data-driven simulations with large-scale patient histology.