The effective Hamiltonian of quantum chromodynamics in the front form of Hamiltonian dynamics is calculated and renormalized. The renormalization group procedure for effective particles up to the second order in the coupling constant is used. Small gluon mass is used to regulate infrared singularities of the theory. The counterterms necessary to renormalize the theory are determined by computing matrix elements of the effective Hamiltonian. The effective Hamiltonians are well-defined symmetric forms on a dense subspace of the Fock space. The zero modes are cut off but, once ultraviolet renormalization is performed, no divergences are found in the color singlet subspace in the limit of the gluon mass approaching zero. A major result is that the interplay between self-energy terms and gluon exchange effective terms generates a term proportional to the quadratic SU(3) Casimir operator times the logarithm of the gluon mass. Therefore, the matrix elements are logarithmically divergent in the color nonsinglet subspace, but finite in the color singlet subspace, because the Casimir operator vanishes in the color singlet subspace. The effective Hamiltonians are suitable for nonperturbative numerical calculations using either classical or quantum computers.