Estimating mutual information

We present two classes of improved estimators for mutual information $M(X,Y)$, from samples of random points distributed according to some joint probability density $\mu (x,y)$. In contrast to conventional estimators based on binnings, they are based on entropy estimates from $k$-nearest neighbor distances. This means that they are data efficient (with $k=1$ we resolve structures down to the smallest possible scales), adaptive (the resolution is higher where data are more numerous), and have minimal bias. Indeed, the bias of the underlying entropy estimates is mainly due to nonuniformity of the density at the smallest resolved scale, giving typically systematic errors which scale as functions of $k∕N$ for $N$ points. Numerically, we find that both families become exact for independent distributions, i.e. the estimator $\widehat{M}(X,Y)$ vanishes (up to statistical fluctuations) if $\mu (x,y)=\mu \left(x\right)\mu \left(y\right)$. This holds for all tested marginal distributions and for all dimensions of $x$ and $y$. In addition, we give estimators for redundancies between more than two random variables. We compare our algorithms in detail with existing algorithms. Finally, we demonstrate the usefulness of our estimators for assessing the actual independence of components obtained from independent component analysis (ICA), for improving ICA, and for estimating the reliability of blind source separation.

Figure 1

Panel (a): Determination of $ϵ\left(i\right)$, $n_x\left(i\right)$, and $n_y\left(i\right)$ in the first algorithm, for $k=1$ and some fixed $i$. In this example, $n_x\left(i\right)=5$ and $n_y\left(i\right)=3$. Panels (b),(c): Determination of ${ϵ}_x\left(i\right)$, ${ϵ}_y\left(i\right)$, $n_x\left(i\right)$, and $n_y\left(i\right)$ in the second algorithm for $k=2$. Panel (b) shows a case in which ${ϵ}_x\left(i\right)$ and ${ϵ}_y\left(i\right)$ are determined by the same point, while panel (c) shows a case in which they are determined by different points.

Figure 2

Estimates of $I^{\left(2\right)}(X,Y)-I_{\text{exact}}(X,Y)$ for Gaussians with unit variance and covariances $r=0.9,0.6,0.3$, and 0.0 (from top to bottom), plotted against $1∕N$. In all cases $k=1$. The number of trials is $>2\times {10}^6$ for $N⩽1000$ and decreases to $\approx {10}^5$ for $N=\mathrm{40\hspace{0.17em}000}$. Error bars are smaller than the sizes of the symbols.

Figure 3

There cannot be any points inside the shaded rectangles. For method 2, this means that the estimates of the marginal entropy $H\left(X\right)\left[H\left(Y\right)\right]$ should be modified, since part of the area outside [inside] the stripe of with ${ϵ}_x$ $\left[{ϵ}_y\right]$ is forbidden. This is neglected in Eq. (9).

Figure 4

Mutual information estimates $I^{\left(1\right)}(X,Y)$ (left panel) and $I^{\left(2\right)}(X,Y)$ (right panel) for Gaussian deviates with unit variance and covariance $r=0.9$, plotted against $k∕N$ (left panel) and $(k-1∕2)∕N$ (right panel), respectively. Each curve corresponds to a fixed value of $N$, with $N=125,250,500,1000,2000,4000,\mathrm{10\hspace{0.17em}000}$, and $\mathrm{20\hspace{0.17em}000}$, from bottom to top. Error bars are smaller than the size of the symbols. The dashed line indicates the exact value $I(X,Y)=\mathrm{0.830\hspace{0.17em}366}$.

Figure 5

Standard deviations of the estimates $I^{\left(1\right)}(X,Y)(+)$ and $I^{\left(2\right)}(X,Y)(\times )$ for Gaussian deviates with unit variance and covariance $r=0.9$, multiplied by $\sqrt{N}$ and plotted against $k\left[I^{\left(1\right)}(X,Y)\right]$ or $k-1∕2\left[I^{\left(2\right)}(X,Y)\right]$. Each curve corresponds to a fixed value of $N$, with $N=125,250,500,1000,2000,4000,\mathrm{10\hspace{0.17em}000}$, and $\mathrm{20\hspace{0.17em}000}$, from bottom to top.

Figure 6

Systematic error $I^{\left(2\right)}(X,Y)-I_{\text{exact}}(X,Y)$ for $k=3$ plotted against $N$ on a log-log scale, for $r=0.9$. The dashed lines are $\propto N^{-0.5}$ and $\propto N^{-0.85}$.

Figure 7

Ratios $I^{\left(2\right)}(X,Y)∕I_{\text{exact}}(X,Y)$ for $k=1$ plotted against $1∕N$, for four different values of $r$.

Figure 8

Statistical errors (one standard deviation) for Gaussian deviates with $r=0.9$, plotted against $N$. Results from $I^{\left(2\right)}(X,Y)$ for $k=1$ (full line) are compared to theoretically predicted (dashed line) and actually measured (dotted line) errors from [8].

Figure 9

Systematic errors for Gaussian deviates with $r=0.0,0.3,0.6$, and $0.9$, plotted against $1∕N$, obtained with the algorithm of [8]. These should be compared to the systematic errors obtained with the present algorithm shown in Fig. 2.

Figure 10

Ratios $I{(X,Y)}_{\text{estim}}∕I_{\text{exact}}(X,Y)$ for the $\Gamma$-exponential distribution, plotted against $1∕N$. These data were obtained with $I^{\left(2\right)}$ using $k=1$, after transforming $x_i$ and $y_i$ to their logarithms. The five curves correspond to $\theta =0.1,0.3,1.0,2.0,10.0$, and 100.0 (from bottom to top).

Figure 11

Ratios $I{(X,Y)}_{\text{estim}}∕I_{\text{exact}}(X,Y)$ for the $\Gamma$-exponential distribution, plotted against $1∕N$. Full lines are from estimator $I^{\left(2\right)}$, dashed lines are from [29]. Our data were obtained with $k=1$ after a transformation to logarithms. The four curves correspond to $\theta =0.1,0.3,2.0$, and 100.0 (from bottom to top for our data, from top to bottom for the data of [29]).

Figure 12

Ratios $I{(X,Y)}_{\text{estim}}∕I_{\text{exact}}(X,Y)$ for the ordered Weinman exponential distribution, plotted against $1∕N$. Full lines are from estimator $I^{\left(2\right)}$, dashed lines are from [29]. Our data were obtained with $k=1$ after a transformation to logarithms. The four curves correspond to $\theta =0.1,0.3,1.0$, and 100.0 (from top to bottom).

Figure 13

Ratios $I^{(1,2)}(X,Y,Z)∕I_{\text{exact}}(X,Y,Z)$ for $k=1$ plotted against $1∕N$, for four different values of $r$. All Gaussians have unit variance and all nondiagonal elements in the correlation matrix ${\sigma }_{i,k},i\ne k$ (correlation coefficients) take the value $r$.

Figure 14

Averages of $I^{(1,2)}\mathbf{(}{X}_1,(X_2\cdots X_m\left)\mathbf{\right)}$ for $k=1$ plotted against $m$ for three different values of $r=0.1,0.5,0.9$. The sample size is $\mathrm{50\hspace{0.17em}000}$; averaging is done over 100 realizations (same parameters as in [30], Fig. 1). Full lines indicate theoretical values, pluses $(+)$ are for $I^{\left(1\right)}$, and crosses $(\times )$ are for $I^{\left(2\right)}$. Squares and dotted lines are read off from Fig. 1 of Ref. 30.

Figure 15

Standard deviations of the estimate $I^{\left(1\right)}$ for Gaussian deviates with unit variance and covariance $r=0.9$, multiplied by $\sqrt{N}∕m$ and plotted against $k$. Each curve corresponds to a fixed value of dimension $m$. Number of samples is $N=\mathrm{10\hspace{0.17em}000}$.

Figure 16

Estimates $I^{\left(1\right)}(m,m^{\prime })-(1∕2)\ln \left[1-C^2(m,m^{\prime })\right]$ for all pairs $(m,m^{\prime })$ of ORFs, plotted against $C^2(m,m^{\prime })$. According to Eq. (A5), this should be positive, which gives an indication of the errors involved in the estimation.

Figure 17

Estimated independent components using JADE.

Figure 18

Left panel: pairwise MIs between all ICA components obtained by JADE, estimated with $I^{\left(1\right)},k=1$. The diagonal is set to zero. Right panel: pairwise MIs between the optimized channels shown in Fig. 19.

Figure 19

Estimated independent components after minimizing $I^1$.

Figure 20

Square roots of variances, $\sqrt{{\sigma }_{ij}}$, of $I^{\left(1\right)}\left[(X_i,X_j)\right]$ (with $k=1$) from JADE output (left panel) and after minimization of $\mathrm{MI}$ (right panel). Again, elements on the diagonal have been set to zero.