Estimation of the entropy based on its polynomial representation

Estimating entropy from empirical samples of finite size is of central importance for information theory as well as the analysis of complex statistical systems. Yet, this delicate task is marred by intrinsic statistical bias. Here we decompose the entropy function into a polynomial approximation function and a remainder function. The approximation function is based on a Taylor expansion of the logarithm. Given $n$ observations, we give an unbiased, linear estimate of the first $n$ power series terms based on counting sets of $k$ coincidences. For the remainder function we use nonlinear Bayesian estimation with a nearly flat prior distribution on the entropy that was developed by Nemenman, Shafee, and Bialek. Our simulations show that the combined entropy estimator has reduced bias in comparison to other available estimators.

Figure 1

Bias and average mean absolute error of various entropy estimators. (a) Average log₂ transformed bias across the [0,log(M)] nats interval (y axis), as a function of the number of observations n (x axis), with the number of states M=16. Black: our new combined Ĥ_comb(n) entropy estimator [Eq. (26)]. Dashed green: our new polynomial estimator T̂(n) [Eq. (10)]. Dashed red: NSB estimator [21]. Red: Bayesian estimator with uniform prior on p [23]. Dashed blue: Grassberger estimator [13]. Dashed-dotted orange: Panzeri-Treves estimator [27]. Magenta: classic Miller-Madow estimator [12]. Cyan: plugin estimator [Eq. (1)]. Dashed black: 10% of the average entropy across the [0,ln(M)] nats interval. (b and c) As in (a), but now for M=81,225. (d) Average mean absolute error of the entropy estimates, averaged across the [0,log(M)] nats interval, with M=16. Legends are similar to (a–c). (e and f): Similar to (d), but now for M=81,225.

Figure 2

(a–c) Entropy estimator bias for various levels of the entropy. Vectors of probabilities were generated by varying the concentration parameter β of a Dirichlet distribution with steps of 1/10 and drawing one vector of probabilities p from each Dirichlet distribution. The x axis corresponds to the entropy of p. The y axis corresponds to the estimator bias in nats. Black: our new combined Ĥ_comb(n) entropy estimator [Eq. (26)]. Dashed green: our new polynomial estimator T̂(n) [Eq. (10)]. Dashed red: NSB estimator [21]. Red: Bayesian estimator with uniform prior on p [23]. Dashed blue: Grassberger estimator [13]. Dashed-dotted orange: Panzeri-Treves estimator [27]. Magenta: classic Miller-Madow estimator [12]. Dashed cyan: plugin estimator [Eq. (1)]. Dashed black: to be estimated entropy of p (true entropy). (a–c) correspond to number of states M=16,81,225 and number of observations n=√M=4, 9, 15, respectively. (d–f) Expected value of various entropy estimators (y axis) as a function of the number of observations n with the expected entropy of p equal to ln(M)−1/1000, that is, close to the maximum entropy. (d–f) correspond to number of states M=16, 81, 225, respectively. (g–i) Similar to (d–f), but now for the expected entropy of p equal to 1/10000, that is, close to the minimum entropy. The Bayesian estimator with uniform prior on p is not shown because of scaling.