Measurements of longitudinal flow decorrelations in 13~TeV $pp$ and 5.44~TeV Xe+Xe collisions with the ATLAS detector are presented. The measurements are performed using the two-particle correlation method, combining charged-particle tracks within $|\eta |<2.5$ with either calorimeter energy clusters or towers within $4.0<|\eta |<4.9$. A template-based subtraction procedure is used to remove non-flow effects in both the $pp$ and the Xe+Xe analyses. The dependence of the longitudinal flow decorrelations on the pseudorapidity separation between the particles is characterized via the slope parameter $F_n$ for the elliptic ($n=2$) and triangular ($n=3$) harmonic moments. The results are reported as a function of charged-particle multiplicity for the $pp$ and Xe+Xe collision systems. Comparing the data to a color string-based model of the initial geometry indicates that in $pp$ and peripheral Xe+Xe collisions, sub-nucleonic structure and fluctuations in longitudinal energy deposition are needed to describe the data.