Fundamental limits on the energy density of the ultralow-frequency primordial gravitational wave background (GWB) radiation which can be obtained from the measurement of secular variations of orbital parameters of binary pulsars are explored. For analytical convenience we choose the simple timing model comprised of the binary system with a pulsar on a circular orbit and a remote observer on the Earth whose motion about the barycenter of the Solar System is assumed to be known with sufficient accuracy. The primordial gravitational waves bring about stochastic noise fluctuations in the times of arrival of the pulsar pulses which include (as our analysis shows) both nonstationary and stationary components. The latter part of the noise is supposed to have spectral power where is the frequency of a gravitational wave intersecting or passing near the line of sight and is the energy density of the GWB radiation. An analytical technique of processing observational data in the time domain is worked out to determine the functional dependence of the pulsar timing residuals and variances of spin and orbital parameters of the binary pulsar with time. This technique enables us to prove that the procedure of fitting the pulsar's spin and orbital parameters acts not only as a low frequency filter of the background noise but also eliminates the nonstationary component of the noise so that the post-fit timing residuals contain only a stationary component. In order to keep the calculations manageable we idealize the observations by assuming that they are uniformly spaced and extend over an integral number of orbital revolutions which is taken so large that any sum over all observation points can be approximated by an integral over the observing period The integrals one meets in calculations are divergent because of the existence of an algebraic singularity in the spectrum of the stochastic gravitational wave background as the frequency approaches the point To avoid this difficulty a powerful method of analytical continuation of Riesz is applied for regularization of all divergent integrals in order to convert them into finite expressions. The regularization procedure enables us to show that the divergent behavior of the spectrum of background gravitational radiation is irrelevant in the treatment of post-fit residuals because of a mutual cancellation of all singular expressions. Our calculations clearly demonstrate that the observed secular variations both of orbital period and of semimajor axis projected onto the line of sight can be effectively used to set a limit on the energy density of the GWB radiation in the frequency range where L is the light travel time from the observer to the pulsar. It is also shown that the upper bound on at the ultralow frequencies of derived by previous authors from observations of PSR B1913+16, was based on an invalid identification of the theoretical expression describing the temporal dependence of the variance of the pulsar's spin frequency derivative with that of its orbital frequency derivative where an overdot denotes a derivative with respect to time. Correct relationships between and variances of either or are derived and used for setting theoretical upper limits on These limits are proportional to until white noise is the dominant source of observational errors. Setting an upper limit on by means of either or measurement cannot be achieved without measuring, at least, two additional post-Keplerian parameters which allow us to determine masses of stars in the binary system and predict the magnitude of and/or The predicted values of these parameters can be compared with their observed values for independent evaluation of variances and/or relating to We apply this strategy for evaluation of and in PSR's B1913+16, B1855+09, B1534+12, and J1713+0747. In particular, this yields an upper limit on in the case of PSR B1913+16 which is not as restrictive as had been previously estimated. A surprising result is that the as yet poorly known estimate of the variance of in the system PSR B1855+09 allows us to set an upper limit on which is less than in the frequency range This limit is already two orders of magnitude more stringent than the one that was previously available but actually invalid. Thus, the precise measurement of appears to be extremely useful in the exploration of properties of the ultralow-frequency gravitational radiation providing a better limit on the cosmological parameter