9.1 Maximum entropy

The principle of maximum entropy selects the distribution that is most consistent with the constraints we know while making the fewest additional assumptions. It justifies many common distributions, including the Gaussian, binomial, and Poisson.

9.1.1 The principle

Given a set of constraints (e.g., known mean and variance), the maximum entropy distribution is the one with the highest entropy among all distributions satisfying those constraints:

$H(p) = -\sum_x p(x) \log p(x)$

Maximize $H(p)$ subject to the constraints. This gives the least informative distribution consistent with what we know — it adds no unjustified assumptions.

# Demonstrate maximum entropy distributions
# For a discrete variable with fixed mean, the maxent distribution is exponential

# Case 1: Fixed mean = 3, values 0:10

x <- 0:10
lambda <- 3 # mean constraint
p_exp <- exp(-x / lambda) / sum(exp(-x / lambda))

cat("Maximum entropy distribution (fixed mean = 3):\n")
cat(" x p(x)\n")
for (i in seq_along(x)) {
cat(sprintf(" %2d %.4f\n", x[i], p_exp[i]))
}

cat(sprintf("\n Mean of distribution: %.3f\n", sum(x _ p_exp)))
cat(sprintf(" Entropy: %.4f\n", -sum(p_exp _ log(p_exp))))

# Compare with a uniform distribution (same support)

p_unif <- rep(1/length(x), length(x))
cat(sprintf("\n Uniform entropy: %.4f\n", -sum(p_unif * log(p_unif))))
cat(" The exponential has lower entropy (more information)\n")
cat(" but satisfies the mean constraint.\n")

# Plot

barplot(p_exp, names.arg = x, col = "steelblue",
main = "Maximum entropy: fixed mean = 3",
xlab = "x", ylab = "p(x)")

Click Run to execute this code in your browser with WebR

9.1.2 The Gaussian as maximum entropy

The Gaussian distribution is the maximum entropy distribution for a continuous variable with known mean and variance. This is why it is the natural default — it assumes nothing beyond the first two moments.

# Demonstrate that the Gaussian is maximum entropy
# for fixed mean and variance

# Compare Gaussian with other distributions with same mean and variance

set.seed(42)
n <- 1e5
mu <- 0
sigma <- 1

# Gaussian

x_gauss <- rnorm(n, mu, sigma)

# Uniform with same mean and variance

# Uniform(a, b): mean = (a+b)/2, var = (b-a)^2/12

a <- mu - sqrt(3) _ sigma
b <- mu + sqrt(3) _ sigma
x_unif <- runif(n, a, b)

# Compute entropy (via histogram approximation)

compute_entropy <- function(x, breaks = 100) {
h <- hist(x, breaks = breaks, plot = FALSE)
p <- h$counts / sum(h$counts)
p <- p[p > 0]
-sum(p * log(p))
}

H_gauss <- compute_entropy(x_gauss)
H_unif <- compute_entropy(x_unif)

cat("Entropy comparison (same mean and variance):\n")
cat(" Gaussian:", round(H_gauss, 4), "\n")
cat(" Uniform: ", round(H_unif, 4), "\n")
cat("\nThe Gaussian has higher entropy\n")
cat("(among smooth distributions with fixed mean and variance)\n")

# Plot

par(mfrow = c(1, 2))
hist(x_gauss, breaks = 80, col = adjustcolor("steelblue", 0.5),
main = "Gaussian", xlab = "x", freq = FALSE, ylim = c(0, 0.45))
curve(dnorm(x, mu, sigma), add = TRUE, lwd = 2, col = "firebrick")
hist(x_unif, breaks = 80, col = adjustcolor("steelblue", 0.5),
main = "Uniform (same mean/var)", xlab = "x", freq = FALSE, ylim = c(0, 0.45))
par(mfrow = c(1, 1))

Click Run to execute this code in your browser with WebR

9.1.3 Binomial and Poisson as maximum entropy

Binomial: Maximum entropy for binary outcomes with a fixed expected count.
Poisson: Maximum entropy for counts with a fixed expected rate.

# Demonstrate binomial and Poisson as maximum entropy
set.seed(42)

# Binomial: n=10, p=0.3

n <- 10
p <- 0.3
x_bin <- 0:n
p_bin <- dbinom(x_bin, n, p)

cat("Binomial distribution (n=10, p=0.3):\n")
cat(" Mean:", n _ p, "\n")
cat(" Entropy:", round(-sum(p_bin _ log(p_bin)), 4), "\n")

# Poisson: lambda = 3

lambda <- 3
x_pois <- 0:20
p_pois <- dpois(x_pois, lambda)

cat("\nPoisson distribution (lambda=3):\n")
cat(" Mean:", lambda, "\n")
cat(" Entropy:", round(-sum(p_pois * log(p_pois)), 4), "\n")

# Plot

par(mfrow = c(1, 2))
barplot(p_bin, names.arg = x_bin, col = "steelblue",
main = "Binomial (n=10, p=0.3)", xlab = "k", ylab = "p(k)")
barplot(p_pois, names.arg = x_pois, col = "firebrick",
main = "Poisson (lambda=3)", xlab = "k", ylab = "p(k)")
par(mfrow = c(1, 1))

Click Run to execute this code in your browser with WebR

Rethinking

Maximum entropy is honest ignorance. The principle of maximum entropy is not about being "objective" or "uninformative" in a philosophical sense. It is about being honest: using only the information you actually have and not pretending to know more than you do. If you know the mean and variance, the Gaussian is the most honest choice. If you know the rate, the Poisson is the most honest choice. Any other distribution would smuggle in assumptions you cannot justify. This is the foundation of the entire GLM framework — the choice of likelihood is not arbitrary; it is the maximum entropy distribution for the type of outcome you are modeling.

Overthinking

Lagrange multipliers and the maxent solution. The maximum entropy distribution is found by maximizing $H(p) = -\sum p_i \log p_i$ subject to constraints $\sum p_i = 1$ and $\sum p_i f_k(x_i) = \bar{f}_k$ . Using Lagrange multipliers:

$\mathcal{L} = -\sum p_i \log p_i - \lambda_0 \left(\sum p_i - 1\right) - \sum_k \lambda_k \left(\sum p_i f_k(x_i) - \bar{f}_k\right)$

Taking the derivative and setting it to zero gives:

$p_i = \frac{1}{Z} \exp\left(-\sum_k \lambda_k f_k(x_i)\right)$

where $Z$ is the normalizing constant. This is an exponential family distribution. The Gaussian, binomial, Poisson, and exponential distributions are all exponential family members — they are all maximum entropy distributions for different constraints.

Summary of Section 9.1

Maximum entropy selects the least informative distribution consistent with known constraints.
Gaussian: maxent for continuous outcomes with known mean and variance.
Binomial: maxent for binary outcomes with known expected count.
Poisson: maxent for counts with known expected rate.
All are exponential family distributions.
Maximum entropy is honest ignorance — use only what you know.

Practice

Generalized linear models