Active matter systems are driven out of thermodynamic equilibrium by localized, microscale energy dissipation. While active field theories are highly successful at simulating these non-equilibrium phenomena (the forward problem), characterizing real-world active materials is fundamentally bottle-necked by the difficulty of measuring active stresses directly. This paper addresses the inverse problem using deep learning: model inference and model selection from observable flow field data of active fluids. We formulate an inversion framework applied to two cornerstone paradigms of active continuum physics: Active Model H (representing scalar active matter) and Active Nematics (representing active systems with orientational order). We demonstrate that the kinetic energy spectrum obtained from the fluid flow field preserves a high-fidelity signature of activity to infer parameters of active model H and active nematics. The classification of the models requires comparison using conventional numerical work against predicted parameters. Our deep learning method presents a principled way to bear upon questions of model inference and selection given the flow field data in continuum models of active matter.