Self-assembled quantum dots (QDs) provide an ideal platform for realization of on-demand maximally entangled photon pairs, which is essential for quantum-information technologies. When the two intermediate exciton states are indistinguishable in energy, the biexciton cascade process naturally yields the maximally entangled photon pairs. However, because of the low symmetry of QDs, the two bright states should belong to different irreducible representations in realistic experiments, and their energy difference, called the “fine-structure splitting” (FSS), is much larger than their homogeneous broadening, giving rise to only classically correlated photon pairs by erasing the “which-way” information. In this work, we show that since these two states belong to different representations, their lifetimes should also be slightly different, which is termed “exciton lifetime asymmetry.” In contrast to the extensively studied FSS, investigation of this lifetime asymmetry is missing in the literature. Here we conduct an investigation of the exciton lifetime asymmetry in self-assembled QDs and present a theory to deduce this asymmetry indirectly from measurable qualities of QDs. We further reveal that the intrinsic lifetimes and their asymmetries are more fundamental quantities of QDs, which determine the upper bound of the extrinsic lifetime asymmetries. Exact relations between lifetime asymmetries, FSS, and the polarization angle are also derived. These relations may be measured in experiments by studying the degree of polarization, as well its evolution under external stress. Our findings provide a complete description of the symmetry of the optical properties of QDs, which can give an important basis to deepen our understanding of QDs.