- Physical Review Journals
All JournalsPhysics Magazine

Physical Review X

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Nematic Order and Orbital Selective Mott State in a Partially Filled Kagome Flat Band

Caiyun Chen1,2,*, Jiangchang Zheng1,*, Yuman He1,*, Siqi Wu1,*, Xuzhe Ying1, Soumya Sankar1, Luanjing Li1, Yizhou Wei1, Xi Dai1 et al.

Hoi Chun Po1,† and Berthold Jäck1,‡

  • *These authors contributed equally to this work.
  • †Contact author: hcpo@ust.hk
  • ‡Contact author: bjaeck@ust.hk
PDF

Phys. Rev. X 16, 011009 – Published 13 January, 2026

DOI: https://doi.org/10.1103/c93j-k53f

Abstract

Coulomb interactions among charge carriers that occupy an electronic flat band (FB) can give rise to captivating phenomena such as quantum criticality, Mott-Hubbard states, and unconventional superconductivity at different FB filling fractions. Consequently, the search for new FB materials with tunable charge carrier filling and strong interactions is a central research theme to broaden our fundamental understanding of strongly interacting quantum systems and to realize materials with novel functionalities. In this work, we present experimental evidence obtained from temperature-dependent scanning tunneling microscopy measurements for various strongly correlated states that appear in partially occupied kagome FBs of Co1−xFexSn whose filling can be controlled by the Fe-doping level x. At elevated temperatures (T≥16  K), we find evidence for rotation symmetry breaking across a broad doping range 0.05<x<0.25. Comparison with model calculations reveals that strong Coulomb interactions (U>100  meV) blend the states of two 3d-orbital-derived FBs and impart a nematic order parameter. Introducing phase-sensitive lock-in measurements as a local scanning tunneling microscopy probe of incompressible states, we also find direct evidence for Hubbard bands in the partially filled kagome flat bands of samples with ideal Fe doping (x=0.17). Results from Hubbard model calculations suggest the presence of an orbital-selective Mott state in a half-filled kagome flat band. Doping-dependent measurements show that the orbital-selective Mott state descends into a correlated metal phase upon electron and hole doping, marked by a sudden collapse of the spectral gap. Our observations demonstrate that the electronic ground state of the kagome FBs sensitively depends on the complex interplay between geometric frustration, strong Coulomb repulsion, orbital degeneracy, and filling fraction at different temperatures. More broadly, our research establishes kagome lattice materials as a unique platform for novel quantum states that arise from strong electronic correlations in a topologically nontrivial FB.

View figure in article

Physics Subject Headings (PhySH)

Popular Summary

In electronic flat bands, the kinetic energy of electrons is strongly reduced, and Coulomb interactions between charge carriers can induce new quantum states, such as Mott insulators and unconventional superconductivity. Focusing on the partially occupied kagome flat bands in Fe-doped CoSn, the researchers utilized temperature-dependent scanning tunneling microscopy (STM) to observe various strongly correlated states appearing at different temperatures and doping levels. Findings indicate that at higher temperatures, the system exhibits nematic order over a wide doping range, induced by strong Coulomb interactions between two different kagome flat bands. The introduction of phase-sensitive measurements with the STM reveals Hubbard bands at ideal iron doping, with model calculations suggesting an orbital-selective Mott state at a half-filled kagome flat band. These insights highlight how geometric frustration, Coulomb repulsion, and orbital degeneracy shape the electronic ground state in kagome lattices, positioning them as promising candidates for exploring new quantum states arising from strong electronic correlations.

Article Text

Supplemental Material

References (64)

  1. A. Bussmann-Holder and H. Keller, High-temperature superconductors: underlying physics and applications, Z. Naturforsch. B 75, 3 (2020).
  2. P. A. Lee, N. Nagaosa, and X.-G. Wen, Doping a Mott insulator: Physics of hightemperature superconductivity, Rev. Mod. Phys. 78, 17 (2006).
  3. B. Keimer, S. A. Kivelson, M. R. Norman, S. Uchida, and J. Zaanen, From quantum matter to high-temperature superconductivity in copper oxides, Nature (London) 518, 179 (2015).
  4. S. M. Girvin, Novel excitations and broken symmetries, in Aspects Topologiques de la Physique en basse Dimension. Topological Aspects of Low Dimensional Systems: Session LXIX. 1998 (Springer, New York, 2002), pp. 53–175.
  5. A. Stern, Anyons and the quantum hall effect—a pedagogical review, Ann. Phys. (Amsterdam) 323, 204 (2008).
  6. Y. Cao, V. Fatemi, S. Fang, K. Watanabe, T. Taniguchi, E. Kaxiras, and P. Jarillo-Herrero, Unconventional superconductivity in magic-angle graphene superlattices, Nature (London) 556, 43 (2018).
  7. Y. Cao, V. Fatemi, A. Demir, S. Fang, S. L. Tomarken, J. Y. Luo, J. D. Sanchez-Yamagishi, K. Watanabe, T. Taniguchi, E. Kaxiras et al., Correlated insulator behaviour at half-filling in magic-angle graphene superlattices, Nature (London) 556, 80 (2018).
  8. Y. Cao, D. Chowdhury, D. Rodan-Legrain, O. Rubies-Bigorda, K. Watanabe, T. Taniguchi, T. Senthil, and P. Jarillo-Herrero, Strange metal in magic-angle graphene with near Planckian dissipation, Phys. Rev. Lett. 124, 076801 (2020).
  9. A. Jaoui, I. Das, G. Di Battista, J. Díez-Mérida, X. Lu, K. Watanabe, T. Taniguchi, H. Ishizuka, L. Levitov, and D. K. Efetov, Quantum critical behaviour in magic-angle twisted bilayer graphene, Nat. Phys. 18, 633 (2022).
  10. N. Regnault and B. A. Bernevig, Fractional chern insulator, Phys. Rev. X 1, 021014 (2011).
  11. Y. Zeng, Z. Xia, K. Kang, J. Zhu, P. Knüppel, C. Vaswani, K. Watanabe, T. Taniguchi, K. F. Mak, and J. Shan, Thermodynamic evidence of fractional chern insulator in moiré MoTe2, Nature (London) 622, 69 (2023).
  12. H. Park, J. Cai, E. Anderson, Y. Zhang, J. Zhu, X. Liu, C. Wang, W. Holtzmann, C. Hu, Z. Liu et al., Observation of fractionally quantized anomalous Hall effect, Nature (London) 622, 74 (2023).
  13. F. Xu, Z. Sun, T. Jia, C. Liu, C. Xu, C. Li, Y. Gu, K. Watanabe, T. Taniguchi, B. Tong et al., Observation of integer and fractional quantum anomalous hall effects in twisted bilayer mote 2, Phys. Rev. X 13, 031037 (2023).
  14. L. Ye, M. Kang, J. Liu, F. Von Cube, C. R. Wicker, T. Suzuki, C. Jozwiak, A. Bostwick, E. Rotenberg, D. C. Bell et al., Massive dirac fermions in a ferromagnetic kagome metal, Nature (London) 555, 638 (2018).
  15. E. Liu, Y. Sun, N. Kumar, L. Muechler, A. Sun, L. Jiao, S.-Y. Yang, D. Liu, A. Liang, Q. Xu et al., Giant anomalous Hall effect in a ferromagnetic kagome-lattice semimetal, Nat. Phys. 14, 1125 (2018).
  16. J.-X. Yin, S. S. Zhang, H. Li, K. Jiang, G. Chang, B. Zhang, B. Lian, C. Xiang, I. Belopolski, H. Zheng et al., Giant and anisotropic many-body spin–orbit tunability in a strongly correlated kagome magnet, Nature (London) 562, 91 (2018).
  17. M. Kang, S. Fang, L. Ye, H. C. Po, J. Denlinger, C. Jozwiak, A. Bostwick, E. Rotenberg, E. Kaxiras, J. G. Checkelsky et al., Topological flat bands in frustrated kagome lattice CoSn, Nat. Commun. 11, 4004 (2020).
  18. W. R. Meier, M.-H. Du, S. Okamoto, N. Mohanta, A. F. May, M. A. McGuire, C. A. Bridges, G. D. Samolyuk, and B. C. Sales, Flat bands in the CoSn-type compounds, Phys. Rev. B 102, 075148 (2020).
  19. B. C. Sales, W. R. Meier, A. F. May, J. Xing, J.-Q. Yan, S. Gao, Y. H. Liu, M. B. Stone, A. D. Christianson, Q. Zhang, and M. A. McGuire, Tuning the flat bands of the kagome metal CoSn with Fe, In, or Ni doping, Phys. Rev. Mater. 5, 044202 (2021).
  20. B. Sales, W. Meier, D. Parker, L. Yin, J. Yan, A. May, S. Calder, A. Aczel, Q. Zhang, H. Li et al., Flat-band itinerant antiferromagnetism in the kagome metal CoSn1−xlnx, Chem. Mater. 34, 7069 (2024).
  21. J.-X. Yin, B. Lian, and M. Z. Hasan, Topological kagome magnets and superconductors, Nature (London) 612, 647 (2022).
  22. H. Huang, L. Zheng, Z. Lin, X. Guo, S. Wang, S. Zhang, C. Zhang, Z. Sun, Z. Wang, H. Weng et al., Flat-band-induced anomalous anisotropic charge transport and orbital magnetism in kagome metal CoSn, Phys. Rev. Lett. 128, 096601 (2022).
  23. S. Sankar, R. Liu, C.-P. Zhang, Q.-F. Li, C. Chen, X.-J. Gao, J. Zheng, Y.-H. Lin, K. Qian, R.-P. Yu et al., Experimental evidence for a berry curvature quadrupole in an antiferromagnet, Phys. Rev. X 14, 021046 (2024).
  24. C. Chen, J. Zheng, R. Yu, S. Sankar, K. T. Law, H. C. Po, and B. Jäck, Visualizing the localized electrons of a kagome flat band, Phys. Rev. Res. 5, 043269 (2023).
  25. L. Chen, F. Xie, S. Sur, H. Hu, S. Paschen, J. Cano, and Q. Si, Emergent flat band and topological kondo semimetal driven by orbital-selective correlations, Nat. Commun. 15, 5242 (2024).
  26. L. Ye, S. Fang, M. Kang, J. Kaufmann, Y. Lee, C. John, P. M. Neves, S. F. Zhao, J. Denlinger, C. Jozwiak et al., Hopping frustration-induced flat band and strange metallicity in a kagome metal, Nat. Phys. 20, 610 (2024).
  27. J. Huang, L. Chen, Y. Huang, C. Setty, B. Gao, Y. Shi, Z. Liu, Y. Zhang, T. Yilmaz, E. Vescovo et al., Non-fermi liquid behaviour in a correlated flat-band pyrochlore lattice, Nat. Phys. 20, 603 (2024).
  28. S. A. Ekahana, Y. Soh, A. Tamai, D. Gosálbez-Martínez, M. Yao, A. Hunter, W. Fan, Y. Wang, J. Li, A. Kleibert et al., Anomalous electrons in a metallic kagome ferromagnet, Nature (London) 627, 67 (2024).
  29. See Supplemental Material at http://link.aps.org/supplemental/10.1103/c93j-k53f, which includes information on the materials characterization and model calculations, additional experimental data to support the findings presented in the main text, and Refs. [30–43].
  30. G. Kresse and J. Furthmüller, Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set, Phys. Rev. B 54, 11169 (1996).
  31. G. Kresse and J. Furthmüller, Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set, Comput. Mater. Sci. 6, 15 (1996).
  32. P. E. Blöchl, Projector augmented-wave method, Phys. Rev. B 50, 17953 (1994).
  33. J. P. Perdew, K. Burke, and M. Ernzerhof, Generalized gradient approximation made simple, Phys. Rev. Lett. 77, 3865 (1996).
  34. A. A. Mostofi, J. R. Yates, G. Pizzi, Y.-S. Lee, I. Souza, D. Vanderbilt, and N. Marzari, An updated version of wannier90: A tool for obtaining maximally-localised wannier functions, Comput. Phys. Commun. 185, 2309 (2014).
  35. G.-X. Zhi, C. Xu, S.-Q. Wu, F. Ning, and C. Cao, Wannsymm: A symmetry analysis code for wannier orbitals, Comput. Phys. Commun. 271, 108196 (2022).
  36. M. F. de Campos, J. C. Damasceno, R. Machado, and C. A. Achete, Uncertainty estimation of lattice parameters measured by x-ray diffraction, in XVIII IMEKO World Conference: Metrology for a Sustainable Development (IMEKO, 2006), see https://www.imeko.org/index.php/proceedings/725-uncertainty-estimation-of-lattice-parameters-measured-by-x-ray-diffraction (2006).
  37. R. á. Feenstra, J. A. Stroscio, and A. á. Fein, Tunneling spectroscopy of the Si(111) 2× 1 surface, Surf. Sci. 181, 295 (1987).
  38. S. Luryi, Quantum capacitance devices, Appl. Phys. Lett. 52, 501 (1988).
  39. J. P. Eisenstein, L. N. Pfeiffer, and K. W. West, Negative compressibility of interacting two-dimensional electron and quasiparticle gases, Phys. Rev. Lett. 68, 674 (1992).
  40. C. J. Chen, Introduction to Scanning Tunneling Microscopy Third Edition (Oxford University Press, USA, 2021), Vol. 69.
  41. H. Shinaoka, J. Otsuki, M. Ohzeki, and K. Yoshimi, Compressing Green’s function using intermediate representation between imaginary-time and real-frequency domains, Phys. Rev. B 96, 035147 (2017).
  42. J. Li, M. Wallerberger, N. Chikano, C.-N. Yeh, E. Gull, and H. Shinaoka, Sparse sampling approach to efficient ab initio calculations at finite temperature, Phys. Rev. B 101, 035144 (2020).
  43. M. Wallerberger, S. Badr, S. Hoshino, S. Huber, F. Kakizawa, T. Koretsune, Y. Nagai, K. Nogaki, T. Nomoto, H. Mori, J. Otsuki, S. Ozaki, T. Plaikner, R. Sakurai, C. Vogel, N. Witt, K. Yoshimi, and H. Shinaoka, sparse-ir: Optimal compression and sparse sampling of many-body propagators, SoftwareX 21, 101266 (2023).
  44. Z. Liu, M. Li, Q. Wang, G. Wang, C. Wen, K. Jiang, X. Lu, S. Yan, Y. Huang, D. Shen et al., Orbital-selective dirac fermions and extremely flat bands in frustrated kagome-lattice metal CoSn, Nat. Commun. 11, 4002 (2020).
  45. Y. Jiang, H. Hu, D. Călugăru, C. Felser, S. Blanco-Canosa, H. Weng, Y. Xu, and B. A. Bernevig, FeGe as a building block for the kagome 1:1, 1:6:6, and 1:3:5 families: Hidden d-orbital decoupling of flat band sectors, effective models, and interaction Hamiltonians, Phys. Rev. B 111, 125163 (2025).
  46. E. H. da Silva Neto, P. Aynajian, R. E. Baumbach, E. D. Bauer, J. Mydosh, S. Ono, and A. Yazdani, Detection of electronic nematicity using scanning tunneling microscopy, Phys. Rev. B 87, 161117(R) (2013).
  47. H. Li, H. Zhao, Q. Yin, Q. Wang, Z. Ren, S. Sharma, H. Lei, Z. Wang, and I. Zeljkovic, Spin-polarized imaging of the antiferromagnetic structure and field-tunable bound states in kagome magnet FeSn, Sci. Rep. 12, 14525 (2022).
  48. J. Tersoff and D. R. Hamann, Theory of the scanning tunneling microscope, Phys. Rev. B 31, 805 (1985).
  49. Y. He, W. Jiang, S. Wu, X. Ying, B. Jäck, X. Dai, and H. C. Po, Interplay of ferromagnetism, nematicity and fermi surface nesting in kagome flat band, arXiv:2510.14593.
  50. I. L. Aleiner, P. W. Brouwer, and L. I. Glazman, Quantum effects in coulomb blockade, Phys. Rep. 358, 309 (2002).
  51. S. Nakatsuji and Y. Maeno, Quasi-two-dimensional mott transition system Ca2−xSrxRuO4, Phys. Rev. Lett. 84, 2666 (2000).
  52. V. Anisimov, I. Nekrasov, D. Kondakov, T. Rice, and M. Sigrist, Orbital-selective mott-insulator transition in Ca2−xSrxRuO4, Eur. Phys. J. B 25, 191 (2002).
  53. M. Vojta, Orbital-selective Mott transitions: Heavy fermions and beyond, J. Low Temp. Phys. 161, 203 (2010).
  54. M. Kang, L. Ye, S. Fang, J.-S. You, A. Levitan, M. Han, J. I. Facio, C. Jozwiak, A. Bostwick, E. Rotenberg et al., Dirac fermions and flat bands in the ideal kagome metal FeSn, Nat. Mater. 19, 163 (2020).
  55. P. A. Lee and T. V. Ramakrishnan, Disordered electronic systems, Rev. Mod. Phys. 57, 287 (1985).
  56. T. Senthil, Critical fermi surfaces and non-fermi liquid metals, Phys. Rev. B 78, 035103 (2008).
  57. M. Yi, D. H. Lu, R. Yu, S. C. Riggs, J.-H. Chu, B. Lv, Z. K. Liu, M. Lu, Y.-T. Cui, M. Hashimoto, S. K. Mo, Z. Hussain, C. W. Chu, I. R. Fisher, Q. Si, and Z. X. Shen, Observation of temperature-induced crossover to an orbital-selective Mott phase in AxFe2−ySe2(A=K,Rb) superconductors, Phys. Rev. Lett. 110, 067003 (2013).
  58. S. Sachdev and B. Keimer, Quantum criticality, Phys. Today 64, No. 2, 29 (2011).
  59. B. Jäck, F. Zinser, E. J. König, Sune N. P. Wissing, A. B. Schmidt, M. Donath, K. Kern, and C. R. Ast, Visualizing the multifractal wave functions of a disordered two-dimensional electron gas, Phys. Rev. Res. 3, 013022 (2021).
  60. K. F. Mak and J. Shan, Semiconductor moiré materials, Nat. Nanotechnol. 17, 686 (2022).
  61. K. P. Nuckolls and A. Yazdani, A microscopic perspective on moiré materials, Nat. Rev. Mater. 9, 460 (2024).
  62. S. Cheng, B. Wang, I. Lyalin, N. Bagués, A. J. Bishop, D. W. McComb, and R. K. Kawakami, Atomic layer epitaxy of kagome magnet Fe3Sn2 and Sn-modulated heterostructures, APL Mater. 10, 061112 (2022).
  63. R. Wiesendanger, Spin mapping at the nanoscale and atomic scale, Rev. Mod. Phys. 81, 1495 (2009).
  64. Data are available under accession code 10.5281/zenodo.11089937.

Outline

Information

Phys. Rev. X 16, 011009– Published 13 January, 2026
  • Received 5 June 2025
  • Revised 18 October 2025
  • Accepted 11 December 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation