Detection of unknown signals in arbitrary noise

We devise a simple method for detecting signals of unknown form buried in any noise, including heavy tailed. The method centers on signal-noise decomposition in rank and time: Only stationary white noise generates data with a jointly uniform rank-time probability distribution, $U(1,N)\times U(1,N)$, for $N$ data points in a time series. Signals of any kind distort this uniformity. Such distortions are captured by rank-time cumulative distributions permitting all-purpose efficient detection, even for single time series and noise of infinite variance.

Figure 1

Proposed method illustrated on a Gaussian pulse $S\left(t\right)=5exp(-4t^2)$ buried in Cauchy noise. (a) Time series of Cauchy noise with four (unknown) Gaussian pulses to detect. Vertical scale is clipped to show detail. (b) The detection metric (see text) is shown, where the four spikes line up with the true pulse locations (red). The detection threshold is marked by the black dashed line. (c) Raw data for the sliding window of $N=41$ centered about the first detected pulse, plotted full scale. (d) The raw data of panel (b) converted to rank from 1 to 41 so, e.g., the sharp negative spike highlighted in gray at left maps to 1 while the large positive outlier in gray at right maps to 41. This collapse of outliers permits the underlying signal to be detected. Ratio of variance or signal-to-noise power ratio is technically zero for the Cauchy noise but the nominal, sample-based value is $\approx 4\times {10}^{-4}$; (e) $p(r,t)$, rank-order probability mass function (pmf) of signal plus noise ($n={10}^6$), with the Gaussian “ridge” seen in the maximum of $p$ vs. $t$. (f) The $\delta \mathcal{C}$ of signal plus noise (see text), nearly exact with $n={10}^6$.

Figure 2

(a) The cdfs of rms values (scaled by their means) of the five-term orthogonal expansion of $\delta \mathcal{C}$ for pure noise: dihedrals $D_4$ and $D_2$, reflections $C_1^{\left(x\right)}$ and $C_1^{\left(y\right)}$, and rotation $R_2$. $C_1^{\left(x\right)}$ and $C_1^{\left(y\right)}$ share a cdf so only four curves are plotted and offset by 1 at the indicated bold tick marks for clarity. The staircases for the coarse discretization of $N=10$ and a single trial $(n=1)$ are surprisingly close to the smooth curves for $(N=320,n=100)$. The $D_4$ cdf is notably broader and, in the case of $N=10$, smoother than the other four. (b) As in Fig. 1, the rms value for the $C_1^{\left(y\right)}$ component of $\delta \mathcal{C}$ for $(N=41,n=1)$ is shown. $C_1^{\left(y\right)}$ works well because the signal is even in time when the sliding window is centered at the peak. There is a noise-induced spike at about $t=500$ in the plotted test statistic (Fig. 1) which lies near the indicated detection threshold, while the weakest pulse near $t=1250$ lies just above that threshold. Seemingly, this suggests that, in a long run, false negatives and positives are commonplace. Not so, as panel (b) shows, by mapping the test statistic to the complement probability. The $log[1-\text{cdf}(x\left)\right]$ approaches a simple asymptote (dashed line), indicating that the cdf itself approaches unity as $exp(-0.0759x^2)$ for $x^2>2.5$. The threshold in Fig. 1, marked by the vertical bar, maps to a probability of ${10}^{-3}$ while the weakest of the four pulses (leftmost red star) is more than 10-fold less likely. Based on the asymptotic expansion, the strongest pulse detection is seen to have a probability $<{10}^{-7}$, well beyond the range of a reasonably accessible Monte Carlo run.

Figure 3

General method of detecting an arbitrary signal of unknown form in any noise: a demonstration on four disparate signals, with $N=256$. Noise is generated via (13) with the color-coded $\nu$ values for all panels as indicated in the legend of panel (a). The optimal symmetry for each signal is shown in the inset. In blue is the signal used for all three DET curves. The same signal form in black [(a)–(c)] shows the amplitude for a DET curve for Gaussian noise of unit variance that matches the Cauchy result ($\nu =1$). Infinite variance of noise for $\nu \le 2$ renders SNR ambiguous but ${\sigma }^2=2$ for $\nu =2$ with SNR values of $[0.113,0.525,0.053,0.888]$ for [(a)–(d)]. (a) The sinc function, $sin\left(x\right)/x$. (b) A chirp signal, e.g., gravity waves recorded by LIGO (Appendix pp3 explains the use of $C_1^{\left(x\right)}$). (c) The Devil's staircase, an example of extreme intermittency, e.g., Ref. [28]. (d) The Weierstrass function with $a=0.99$ and $b=6$, an example of a continuous but nowhere differentiable signal. The second inset (upper left) shows all 16 traces for the case of $\nu =1$. The vertical scale is clipped drastically down to signal level, yet none is discernible by eye. Overlain on the blue inset signal is the mean rank (black dots), linearly rescaled to match the signal. Note how all 10 prominent signal peaks map to extremal values of mean rank (circles).

Figure 4

Detection and extraction of a weak linear signal buried in noise: Even for single realizations ($n=1$), the proposed method is nearly as efficient for heavy-tailed as for normal noise. The signal $f\left(x\right)=\alpha x/N$ is sampled uniformly at $N$ points on $[-1,1]$. Three noise distributions are used: $\nu =1$ (Cauchy noise, blue), $\nu =2$ (red), and $\nu \to \infty$ (Gaussian noise, gold), with solid lines for the proposed method and broken lines for conventional methods: dashed for least squares (LS) and dotted for Theil-Sen. (a) The means and standard deviations of $\overline{\delta \mathcal{C}}$ determine performance of the detection metric $\tau$, a scaled difference of means (Welch statistic), defined in the legend, with subscripts 1 for signal absent and 2 for signal present. The ln-ln plot of $\tau$ vs. the (scaled) slope $\alpha$ shows $\tau =\alpha +c$ for all cases plotted, varying only in the intercept $c$. For normal noise, LS is slightly better. For Cauchy noise, the present method edges out Theil-Sen everywhere except for $\alpha >3$. (b) Detection error trade-off curves for $\alpha =40$. Operating at $P_{\mathrm{FA}}=P_{\mathrm{MD}}$, detection error ranges from 20% at $\nu =1$ to 5% as $\nu \to \infty$. The companion broken curves are almost identical. (c) Slope extraction for $f\left(x\right)=exp\left(1\right)x/48\approx 0.0566x$. We determine slope by requiring $\overline{\delta \mathcal{C}}=0$ (see text) and match the Theil-Sen histogram as well as the narrower LS result.