Recently, the superconducting diode effect, characterized by unequal critical currents in opposite directions, has been observed experimentally and predicted theoretically in models of bulk superconductors and Josephson junctions. In this work, we construct a Josephson junction using a recently discovered unconventional coplanar magnet, the $p$-wave magnet, with proximity-induced superconductivity, and demonstrate the emergence of a field free Josephson diode effect (JDE). The barrier region is formed by another unconventional collinear magnet, namely an altermagnet. We show that in addition to time-reversal and inversion symmetries, breaking the mirror symmetry $M_{yz}$ is essential for the JDE in our geometry, and that the unconventional-magnet heterostructure provides a platform to isolate this symmetry requirement. Unlike earlier models that realize JDE using unconventional magnets, this setup does not require Rashba spin–orbit coupling or different superconductors across the junction. Instead, $p$-wave magnet provides inversion-asymmetric spin splitting while the altermagnetic barrier breaks mirror symmetry, together giving rise to the diode response. Finally, we establish the robustness of the JDE against interface mismatch, disorder, finite-temperature effects, and variations in the induced superconducting gap over a broad parameter regime, making the proposed heterostructure a promising platform for field-free superconducting diodes.