Signals as departures from random walks

We study statistics of data ranking, focusing on the recently discovered distribution-invariant discrete eigenvalue spectrum for an independent and identically distributed (IID) process. We employ a variant of a cumulative distribution function in rank and time that maps the sampling variability for an IID process onto a set of random walks. This mapping admits confidence bounds on whether the residual (data with signal removed) arises solely from IID sampling variability. Any deviations judged significant are regarded as signals, whether deterministic, chaotic, or random. Unlike our recent work on extracting unknown signals in arbitrary noise, here we focus on aspects that are easily combined with any other methods of signal extraction. The ubiquitous case of a single trace receives particular attention. The approach is illustrated on dark current and gamma-ray arrival datasets where we examine the residual for consistency with the expected sampling variability of IID noise.

Figure 1

Ranking patterns distinguish variants of white noise. [(a)–(d)] Four traces of length $N=32$. Panel (a) is an output of a strict (IID) Gaussian white noise whereas (b)–(d) are white ($\delta$ correlated) but not IID. Panel (b) (“strong but not strict white noise”) is an uncorrelated but dependent random variable $z_j\equiv x_j{x}_{j-1}$, with IID $x$ drawn from $\mathcal{N}(0,1)$. Panel (c) (“white chaos”) is the output of logistic map $x_{j+1}=a{x}_j(1-x_j)$, uncorrelated at the chosen $a=4.0$. Panel (d) (“wide-sense stationary”) is from a Pearson pdf with a quadratically varying kurtosis. The four data traces look similar but panels (e)–(h), paired with (a)–(d) and based on $n={10}^7$ realizations each, differ in standard deviation of rank patterns. (e) The mean value of ${\sigma }_{\mathrm{rank}}$ for IID noise for equally likely integers $1,2,...,32$ is given by $\sqrt{(N-1)(N+1)/12}=\sqrt{341}/2\approx 9.23$. However, despite the whiteness, this statistically uniform ${\sigma }_{\mathrm{rank}}$ profile for IID noise is disturbed by the spikes in (f) and (g) at $j=1,32$ (see Appendix pp1 for details). (h) ${\sigma }_{\mathrm{rank}}$ follows the subtle kurtosis pattern, despite the constant variance of the data itself.

Figure 2

A new distribution-invariant description of an IID discrete random process: joint rank-time empirical cumulative distribution. (a) Magnitude $f\left(t\right)$ of a log-normally fluctuating quantity vs time $t$. To emphasize the general setting, $f\left(t\right)$ is measured in arbitrary units and sampled unevenly. (b) Rank $r$ vs time $t$ for the data of panel (a). Monotonic transformation of data, inset $F\left(f\right(t\left)\right)$, yields the same $r$ vs $t$ as rank depends solely on serial order. Nor does the nonuniform sampling in time affect the abscissa as the time index is also integer valued from 1 to 10. (c) Time vs rank for the data of panel (a): $t$ and $r$ are statistically indistinguishable for a discrete IID process. (d) Joint rank-time histogram, whose normalized version is a probability mass function (pmf). (e) Joint deviation cumulative distribution function (cdf), $\delta \mathcal{C}$, defined in Eq. (1), i.e., the difference between the actual and theoretical IID cdf, for data of panel (d), designed to contrast the actual sampling variability with that of an IID process. (f) Same as panel (d) but extending the dataset in (a) to $n=100$, yielding a histogram closer to the ensemble limit pmf $U(1,N)\times U(1,N)$. (g) Relative fluctuations of $\delta \mathcal{C}$, similar to those in (e) but with the decrease in amplitude by a factor of $\sqrt{n}=10$. For any $n$, each row and column slice of the augmented $\delta \mathcal{C}$ is a pinned random walk (see text).

Figure 3

Rank and time slices of IID-based sample $\delta \mathcal{C}$ [empirical deviation cdf; see Eq. (1)] are pinned random walks. (a) A 128 pixel sample of dark current (IID proxy), recorded as the raw output of a 14 bit A-D converter from 1 s exposure on a Canon R5 camera, in the absence of incident photons (lid closed). (b) $\delta \mathcal{C}$ calculated from panel (a) data. (c) The middle row and column slices of panel (b) $\delta \mathcal{C}$: rank and time are statistically equivalent for IID processes. Panels (d)–(f) parallel the top three panels but for a 128 pixel segment taken from a different row of the image. As illustrated in panels (c) and (f), all IID-based $\delta \mathcal{C}$ slices (rows and columns) execute pinned random walks (see text). The universal eigenfunction expansion (see goodness of fit section) shows that lowest frequency mode [“half wave,” red dashed curves in panel (c)] is prominent, accounting for about 61% and 37% of the variance in 1D and 2D IID variability, respectively. In panel (f), the $\delta \mathcal{C}$ slices for panel (d) project most on the second and third mode (on average 15.2% and 6.8% of 1D variance, respectively). For illustrative purposes, traces in panels (c) and (f) were picked among several hundreds for the closest resemblance to pure modes.

Figure 4

An agnostic criterion for judging fits to data: To what confidence level is the residual IID? (a) Simulated counts of $\gamma$-ray photons, received at the BATSE instrument [21], (apj457785), with three constant rate intervals marked by red and durations determined in [21] via Bayesian change point analysis. As ranking is unaffected by a constant offset, either one or two rate constants are to be fitted. (b) A single parameter to fit ($d=1$) the middle excess rate relative to the first segment in (a) is subtracted. The residual is shown in red. (c) Data in (a) after removing two excess rates ($d=2$) as in [21], now second and third segments shown in red. (d) Top view (color map) of the deviation cdf $\delta \mathcal{C}$ for the raw data in (a), with a pronounced “hill-valley” pattern, signaling a strong deviation from IID. (e) $\delta \mathcal{C}$ for the data in (b) (same color range) and no clear pattern. (f) Same as (e) but for data in (c) and no pattern evident. (g) Values of $\delta {\mathcal{C}}_{\mathrm{rms}}$ are overlaid on the IID distribution to answer the question: How likely is a given trace to have arisen from pure noise? Raw data $\delta {\mathcal{C}}_{\mathrm{rms}}=0.0208$ are exceedingly unlikely, but which is “better”: the $d=1$ or $d=2$ fit? (In both cases, the residual data are white as judged by $\delta$ correlation and a flat FFT.) (h) The $d=2$ residual results in a rather small $\delta {\mathcal{C}}_{\mathrm{rms}}$, exceeded by pure noise 98% of the time whereas the $d=1$ fit $\delta {\mathcal{C}}_{\mathrm{rms}}$ residual variance is exceeded by 9.1% of IID realizations (see inset table). The insets are rank-time color maps (top view) of the outstanding modes of $\delta \mathcal{C}$ and the associated signal patterns in time. These results suggest that two parameters are not needed. The new expansion (see text) pinpoints oversuppression of the second mode (darker gray bar) as the cause.

Figure 5

Ranking of the logistic map $x_{k+1}=4x_k(1-x_k)$ for $N=3$ (ranks 1,2,3). (a) Three domains where each of the consecutive variables $\{x_1,x_2,x_3\}$ is the smallest (rank 1). (b) The probability that $x_1$ is rank 1: ${\int }_0^{5/8-\sqrt{5}/8}p\left(x\right)dx=2/5$. Integrating from $3/4$ to 1 gives a probability of $1/3$ for $x_2$ to be rank 1, and the middle third yields $4/15$ for the probability that $x_3$ is the lowest rank. Note that the middle point $x_2$ attains the probability expected for all three time slots with an IID process. Hence, the departure from uniformity is an “end effect.” The three areas in panel (b) show that a symmetrical scheme for rank would obtain were the first border moved back to $1/4$, thereby yielding three equal probabilities of $1/3$.