- Letter
Kitaev interaction and possible spin liquid state in and
Phys. Rev. B 112, L220412 – Published 26 December, 2025
DOI: https://doi.org/10.1103/c6r4-fgj5
Abstract
Kitaev materials are of great interest due to their potential in realizing quantum spin liquid (QSL) states and applications in topological quantum computing. In the pursuit of realizing Kitaev QSL, a Mott insulator with strong bond-dependent frustration and weak geometric frustration is highly desirable. Here we explore Kitaev physics in the van der Waals triangular antiferromagnet (AF) , through the spin-orbital states and Wannier function analyses, exact diagonalization and density matrix renormalization group study of the electronic structure and magnetic properties. We find that the high-spin ion is in the state because of strong spin-orbit coupling, and the weak trigonal elongation and crystal field contribute to the observed weak in-plane magnetic anisotropy. The strong hopping via the strong Co -I hybridization gives rise to a strong Kitaev interaction () at the first nearest neighbors (1NN), and the long Co-Co distance and the weak hoppings determine a weak Heisenberg interaction . The resultant = 6.63 confirms a strong bond-dependent frustration, while the geometric frustration due to the 3NN Heisenberg interaction gets involved, and they all together result in the experimental helical AF order in . We then propose to suppress the using a partial Mg substitution for Co, and indeed we find that has the much reduced geometric frustration but hosts the robust bond-dependent frustration, and thus it would be a promising Kitaev material being so far closest to the QSL state.