Calculating the entropy of the Poisson distribution

The entropy of a Poisson distribution has no closed form. The entropy, in nats, is:

$$ \begin{aligned} H(\lambda) &= - \sum_{k=0..\infty} \Pr(k) \log\Pr(k) \\ &= - \sum_{k=0..\infty} \frac {\lambda^k e^{-\lambda}}{k!} \left[ k \log(\lambda) - \lambda - \log(k!) \right] \\ &= \lambda[1 - \log(\lambda)] + e^{-\lambda}\sum_{k=0..\infty} \frac{\lambda^k\log(k!)}{k!} \end{aligned} $$

These notes evaluate some numerical approximations to the entropy.

For high firing rates, the Poisson distribution becomes approximately Gaussian with $\mu=\sigma^2=\lambda$. Using the forumula for the entropy of the Gaussian, this implies

$$ H(\lambda) = \tfrac 1 2 \log(2 \pi e \lambda) + \mathcal{O}\left(\tfrac 1 \lambda \right) $$

Compare this to the fist few terms of the series expression, which is increasingly accurate for large $\lambda$, but diverges for small $\lambda$:

$$ H(\lambda) = \frac{1}{2}\log(2 \pi e \lambda) - \frac{1}{12 \lambda} - \frac{1}{24 \lambda^2}-\frac{19}{360 \lambda^3} + O\left(\frac{1}{\lambda^4}\right) $$

Another way to calculate the entropy for low rates is calculate it using a finite number of terms in the series expansion. For low rates, the probability of $k> \lambda + 4\sqrt{\lambda}$ events is neglegible, so we only need to sum a small number of terms.

$$ H(\lambda) \approx - \textstyle\sum_{k=0}^{\lceil \lambda + 4\sqrt{\lambda} \rceil} \frac {\lambda^k e^{-\lambda}}{k!} \left[ k \log(\lambda) - \lambda - \log(k!) \right] $$

Numerically, you can use the above calculation for $\lambda<1.78$, taking terms out to $k<\lceil \lambda + 4\sqrt{\lambda} \rceil$. For $\lambda\ge 1.78$, the third-order approximation for large $\lambda$ becomes accurate.

$$ H(\lambda) \approx \begin{cases} - \textstyle\sum_{k=0}^{\lceil \lambda + 4\sqrt{\lambda} \rceil} \frac {\lambda^k e^{-\lambda}}{k!} \left[k \log(\lambda) - \lambda - \log(k!)\right] & \lambda<1.78\\ \frac{1}{2}\log(2 \pi e \lambda) - \frac{1}{12 \lambda} - \frac{1}{24 \lambda^2}-\frac{19}{360 \lambda^3} & \lambda\ge 1.78 \\ \end{cases} $$

The relative error in this approximation is $|\hat H-H|/|H|<0.0016$