We studied the structural and magnetic properties of the double perovskite oxides containing mercury, ( = Hg, , and Ca). and ( crystallize in a rhombohedral phase with space group, while crystallizes in a monoclinic phase with space group. The magnetic susceptibility, specific heat, and neutron diffraction measurements show that, upon cooling, the commensurate antiferromagnetic (AFM) order occurs at 10 K, 8 K, and 11 K for = Hg, , and Ca systems, respectively. The first-principles calculations agree well with the experimental results, showing that the type-I AFM spin arrangement, i.e., magnetic moments aligned ferromagnetically within the planes and antiferromagnetically between adjacent planes, has the lowest total energy compared to other types of long-range order. For all three systems, the magnetic entropy associated with the AFM order amounts to less than 80% of the theoretical value for the high-spin state of , indicating magnetic frustration across the series. Additionally, an incommensurate order at 5 K was identified by neutron diffraction for , which can be described by helicoidal spin order. This observation makes one of the most complex magnetic systems among double perovskite oxides.