Materials underpin modern technologies, from energy harvesting, storage, and conversion to information and communication technologies. Their functionality is often governed by the interplay between competing phases, and thermodynamic behavior shapes microscopic properties and ultimately determines technological performance; for instance, the light absorption of inorganic metal-halide perovskites in solar cells. Accurately predicting crystal thermodynamics, however, remains a major challenge for computational approaches because strong anharmonic effects require extensive sampling of the potential energy surface. Here, we present an on-the-fly Bayesian framework, combined with the stochastic self-consistent harmonic approximation, for learning first-principles interatomic potentials. This approach enables the prediction of thermodynamic properties over a broad temperature range with first-principles accuracy while requiring training on only a few tens to a few hundreds of atomic configurations. A relevant finding is that the majority of the required structures are obtained from a single inexpensive dataset generation step, demonstrating that the relevant physics for accurately predicting solid-state thermodynamics is encoded in a few physically meaningful configurations. To demonstrate its power, we investigate the thermodynamic and dynamical properties of Li${}_2$O, $\alpha$-CsPbI${}_3$, and $\delta$-CsPbI${}_3$, requiring only 44, 256, and 50 total-energy calculations, respectively. Notably, we show that this framework accurately captures the extreme thermal expansion of Li${}_2$O, and the phase diagram of CsPbI${}_3$, which explains its spontaneous degradation into the non-absorbing yellow phase, with a predicted transition temperature within 20 K of experiment. More broadly, the method presented opens an interesting route toward accelerated materials engineering under realistic conditions for a wide range of technologically relevant applications, including solid-state batteries, optoelectronic devices, and memristors.