Quantum Conference Key Agreement (CKA) provides a secure method for multi-party communication. A recently developed interference-based prepare-and-measure quantum CKA possesses the advantages of measurement-device-independence, namely, being immune to side-channels from the detector side. Besides, it achieves good key rate performance, especially for high-loss channels, due to the use of single photon interference. Meanwhile, several fully passive QKD schemes have been proposed, which eliminate all side channels from the source modulation side. We extend the fully passive idea to an interference-based CKA, which has a high level of implementation security for many-user communication.