9.3 Maximum entropy priors

Maximum entropy can also guide the choice of prior distributions. A maximum entropy prior is the least informative prior compatible with what we actually know, preventing us from smuggling in unjustified assumptions.

9.3.1 The principle for priors

When choosing a prior, we should use the maximum entropy distribution consistent with what we know:

$p(\theta) = \arg\max_{q} H(q) \quad \text{s.t.} \quad \mathbb{E}_q[f(\theta)] = \bar{f}$

If we know only the range $[a, b]$ : uniform prior (maximum entropy on a bounded interval).
If we know only the mean: exponential prior (maximum entropy on $[0, \infty)$ with fixed mean).
If we know mean and variance: Gaussian prior (maximum entropy on $\mathbb{R}$ with fixed moments).

# Demonstrate maximum entropy priors
# Compare different priors for a probability parameter

# Uniform prior (maxent for known range only)

p_grid <- seq(0.001, 0.999, length.out = 200)
prior_uniform <- dunif(p_grid, 0, 1)

# Beta(1, 1) = uniform

prior_beta11 <- dbeta(p_grid, 1, 1)

# Beta(2, 2) (slightly informative, symmetric)

prior_beta22 <- dbeta(p_grid, 2, 2)

# Beta(5, 5) (more informative)

prior_beta55 <- dbeta(p_grid, 5, 5)

# Compute entropies (via numerical integration)

compute*entropy_continuous <- function(p, x) {
dx <- x[2] - x[1]
p <- p / sum(p * dx) # normalize
-sum(p _ log(p) * dx)
}

cat("Entropy of priors:\n")
cat(" Uniform [0,1]:", round(compute_entropy_continuous(prior_uniform, p_grid), 4), "\n")
cat(" Beta(1,1): ", round(compute_entropy_continuous(prior_beta11, p_grid), 4), "\n")
cat(" Beta(2,2): ", round(compute_entropy_continuous(prior_beta22, p_grid), 4), "\n")
cat(" Beta(5,5): ", round(compute_entropy_continuous(prior_beta55, p_grid), 4), "\n")

cat("\nUniform has the highest entropy (least informative)\n")
cat("More concentrated priors have lower entropy\n")

# Plot

plot(p_grid, prior_uniform, type = "l", lwd = 2, col = "steelblue",
main = "Maximum entropy priors for a probability",
xlab = "p", ylab = "density", ylim = c(0, max(prior_beta55) * 1.1))
lines(p_grid, prior_beta22, lwd = 2, col = "firebrick")
lines(p_grid, prior_beta55, lwd = 2, col = "darkgreen")
legend("top", legend = c("Uniform / Beta(1,1)", "Beta(2,2)", "Beta(5,5)"),
col = c("steelblue", "firebrick", "darkgreen"), lwd = 2)

Click Run to execute this code in your browser with WebR

9.3.2 When to use maxent priors

Maximum entropy priors are appropriate when:

You have genuine ignorance about the parameter.
You want to avoid smuggling in unjustified assumptions.
You need a default prior that is principled, not arbitrary.

However, when you have real prior information (from previous studies, theory, or domain knowledge), you should use it. The maximum entropy principle is a baseline, not a constraint.

# Demonstrate the effect of prior choice on the posterior
# Model: y ~ Bernoulli(p), observe 3 successes in 10 trials

set.seed(42)
n <- 10
successes <- 3
failures <- n - successes

p_grid <- seq(0.001, 0.999, length.out = 500)

# Different priors

priors <- list(
"Uniform" = dunif(p_grid, 0, 1),
"Beta(2,2)" = dbeta(p_grid, 2, 2),
"Beta(5,5)" = dbeta(p_grid, 5, 5),
"Beta(1,10)" = dbeta(p_grid, 1, 10) # skeptical prior
)

# Compute posteriors

cat("Effect of prior on posterior (3/10 successes):\n")
for (name in names(priors)) {
prior <- priors[[name]]
likelihood <- dbinom(successes, n, p_grid)
posterior <- likelihood * prior
posterior <- posterior / sum(posterior)

# Posterior mean

post_mean <- sum(p_grid * posterior)

# 95% CI

cum <- cumsum(posterior) / sum(posterior)
lower <- p_grid[which(cum >= 0.025)[1]]
upper <- p_grid[which(cum >= 0.975)[1]]

cat(sprintf(" %-12s: posterior mean = %.3f, 95%% CI = [%.3f, %.3f]\n",
name, post_mean, lower, upper))
}

cat("\nAll priors give similar posteriors with enough data.\n")
cat("With little data, the prior matters more.\n")

Click Run to execute this code in your browser with WebR

Rethinking

Priors are not subjective — they are assumptions. The maximum entropy principle gives us a principled way to choose priors that encode only what we know. But "what we know" is itself a judgment. If we know the parameter is bounded, we should use a bounded prior. If we know it is positive, we should use a positive prior. If we know nothing, we should use the maximum entropy prior. The key is to be explicit about what we know and let the prior reflect it. A flat prior on $[-\infty, \infty]$ is not "objective" — it is a claim that all values are equally plausible, which is rarely true.

Summary of Section 9.3

Maximum entropy priors encode only what we know — no more, no less.
Known range → uniform; known mean → exponential; known mean and variance → Gaussian.
Maxent priors are a baseline for genuine ignorance.
When you have real prior information, use it — maxent is not a constraint.
Priors are assumptions — be explicit about what you know.

Generalized linear models

Summary