6.2 Information theory and model performance

Information theory provides the language for measuring model performance. The key quantity is the expected log-probability, which measures how surprised the model would be by new data. Lower surprise means better prediction.

6.2.1 Entropy

The entropy of a distribution measures its uncertainty:

$H(p) = -\sum_{i} p_i \log p_i$

Entropy is maximized when all outcomes are equally likely (maximum uncertainty) and minimized when one outcome has probability 1 (no uncertainty).

# Compute entropy for different distributions
# Entropy: H(p) = -sum(p * log(p))

# Fair coin: maximum entropy

p_fair <- c(0.5, 0.5)
H_fair <- -sum(p_fair * log(p_fair))
cat("Fair coin entropy:", round(H_fair, 4), "\n")

# Biased coin: lower entropy

p_biased <- c(0.9, 0.1)
H_biased <- -sum(p_biased * log(p_biased))
cat("Biased coin entropy:", round(H_biased, 4), "\n")

# Three outcomes

p_three <- c(0.7, 0.2, 0.1)
H_three <- -sum(p_three * log(p_three))
cat("Three outcomes (0.7, 0.2, 0.1) entropy:", round(H_three, 4), "\n")

# Uniform three outcomes: maximum entropy

p_uniform <- c(1/3, 1/3, 1/3)
H_uniform <- -sum(p_uniform * log(p_uniform))
cat("Uniform three outcomes entropy:", round(H_uniform, 4), "\n")

cat("\nEntropy is maximized when all outcomes are equally likely.\n")

Click Run to execute this code in your browser with WebR

6.2.2 Kullback-Leibler divergence

The Kullback-Leibler (KL) divergence measures how different two distributions are:

$D_{\text{KL}}(p \| q) = \sum_i p_i \log \frac{p_i}{q_i}$

The KL divergence is always non-negative and is zero only when $p = q$ . It measures the information lost when $q$ is used to approximate $p$ .

# Compute KL divergence
# D_KL(p || q) = sum(p * log(p/q))

# True distribution

p <- c(0.7, 0.2, 0.1)

# Two approximations

q1 <- c(0.6, 0.3, 0.1) # close to p
q2 <- c(0.3, 0.3, 0.4) # far from p

KL1 <- sum(p _ log(p / q1))
KL2 <- sum(p _ log(p / q2))

cat("D_KL(p || q1) =", round(KL1, 4), "(close approximation)\n")
cat("D_KL(p || q2) =", round(KL2, 4), "(poor approximation)\n")
cat("\nKL divergence is always >= 0\n")
cat("KL = 0 only when p = q\n")

Click Run to execute this code in your browser with WebR

6.2.3 Log-likelihood as a measure of model performance

The expected log-likelihood (or expected log-pointwise predictive density) is the natural measure of model performance:

$\text{lppd} = \sum_{i=1}^{N} \log E_{\theta \mid D}[p(y_i \mid \theta)]$

This measures how well the model predicts new data. The higher the lppd, the better the model. The connection to information theory is through the KL divergence: the model that minimizes $D_{\text{KL}}(p_\text{true} \| p_\text{model})$ also maximizes the expected log-likelihood.

# Demonstrate log-likelihood as model performance
set.seed(42)
n <- 100
x <- runif(n, 0, 10)
y <- 1 + 2 * x + rnorm(n, sd = 2)

# Three models with different complexity

m1 <- lm(y ~ x) # linear (correct)
m2 <- lm(y ~ x + I(x^2)) # quadratic (overfit)
m3 <- lm(y ~ 1) # intercept only (underfit)

# Log-likelihood (higher is better)

ll1 <- logLik(m1)
ll2 <- logLik(m2)
ll3 <- logLik(m3)

cat("Log-likelihoods (higher is better):\n")
cat(" Linear (correct):", round(as.numeric(ll1), 2), "\n")
cat(" Quadratic (overfit):", round(as.numeric(ll2), 2), "\n")
cat(" Intercept only (underfit):", round(as.numeric(ll3), 2), "\n")

cat("\nThe quadratic model has the highest log-likelihood\n")
cat("because it has more parameters. But it may overfit.\n")
cat("Information criteria adjust for this.\n")

Click Run to execute this code in your browser with WebR

Rethinking

Information theory unifies model comparison. The KL divergence provides a principled way to compare models: the best model is the one that minimizes the KL divergence from the true distribution. Since we do not know the true distribution, we cannot compute the KL divergence directly. But we can estimate it using information criteria (Section 6.4), which approximate the expected log-likelihood on new data. This is the information-theoretic foundation of model comparison.

Overthinking

From KL divergence to WAIC. The connection between KL divergence and information criteria is as follows. The expected log-likelihood on new data is:

$E_{\tilde{y}}[\log p(\tilde{y} \mid D)] = E_{\tilde{y}}\left[\sum_i \log \int p(\tilde{y}_i \mid \theta) \, p(\theta \mid D) \, d\theta\right]$

This is the out-of-sample lppd. We cannot compute it directly because we do not have new data. But we can estimate it from the sample. The in-sample lppd is biased upward (overfitting), so we need a penalty. WAIC uses the variance of the log-likelihood across posterior samples as the penalty:

$\text{WAIC} = -2(\text{lppd} - p_\text{WAIC})$

where $p_\text{WAIC} = \sum_i \text{Var}_{\theta \mid D}[\log p(y_i \mid \theta)]$ is the effective number of parameters.

Summary of Section 6.2

Entropy $H(p) = -\sum p_i \log p_i$ measures uncertainty.
KL divergence $D_{\text{KL}}(p \| q)$ measures the information lost when $q$ approximates $p$ .
Expected log-likelihood is the natural measure of model performance.
The best model minimizes KL divergence from the truth, which maximizes expected log-likelihood.
WAIC estimates out-of-sample lppd with a penalty for effective parameters.

The problem with parameters

Regularization