6.4 Information criteria

Information criteria such as WAIC and PSIS estimate the expected out-of-sample performance of a model. They combine the log-predictive density with a penalty for the effective number of parameters.

6.4.1 WAIC

The Widely Applicable Information Criterion (WAIC) estimates out-of-sample deviance:

$\text{WAIC} = -2(\text{lppd} - p_\text{WAIC})$

where:

lppd (log pointwise predictive density) = $\sum_i \log \frac{1}{S} \sum_s p(y_i \mid \theta_s)$
$p_\text{WAIC}$ (effective number of parameters) = $\sum_i \text{Var}_s[\log p(y_i \mid \theta_s)]$

# Compute WAIC manually
set.seed(42)
n <- 100
x <- runif(n, 0, 10)
y <- 1 + 2 * x + rnorm(n, sd = 2)

# Fit two models

m1 <- lm(y ~ x) # linear (correct)
m2 <- lm(y ~ x + I(x^2)) # quadratic (slightly overfit)

# Compute AIC for comparison

cat("AIC comparison:\n")
cat(" Linear:", round(AIC(m1), 2), "\n")
cat(" Quadratic:", round(AIC(m2), 2), "\n")
cat(" Difference:", round(AIC(m2) - AIC(m1), 2), "\n")

# Compute WAIC manually

# For Gaussian model: log-likelihood per observation

library(MASS)

compute_waic <- function(fit, y, x) {
post <- mvrnorm(1000, mu = coef(fit), Sigma = vcov(fit))
sigma <- summary(fit)$sigma

# Log-likelihood matrix: [S x N]

ll <- matrix(0, nrow = 1000, ncol = length(y))
for (s in 1:1000) {
if (ncol(post) == 2) {
mu <- post[s, 1] + post[s, 2] _ x
} else {
mu <- post[s, 1] + post[s, 2] _ x + post[s, 3] * x^2
}
ll[s, ] <- dnorm(y, mean = mu, sd = sigma, log = TRUE)
}

# lppd

lppd <- sum(log(colMeans(exp(ll))))

# p_WAIC

p_waic <- sum(apply(ll, 2, var))

# WAIC

waic <- -2 * (lppd - p_waic)

return(list(waic = waic, lppd = lppd, p_waic = p_waic))
}

w1 <- compute_waic(m1, y, x)
w2 <- compute_waic(m2, y, x)

cat("\nWAIC comparison:\n")
cat(" Linear: WAIC =", round(w1$waic, 2),
  " (lppd =", round(w1$lppd, 2), ", p_WAIC =", round(w1$p_waic, 2), ")\n")
cat("  Quadratic: WAIC =", round(w2$waic, 2),
" (lppd =", round(w2$lppd, 2), ", p_WAIC =", round(w2$p_waic, 2), ")\n")
cat(" Difference:", round(w2$waic - w1$waic, 2), "\n")

Click Run to execute this code in your browser with WebR

6.4.2 PSIS and leave-one-out cross-validation

PSIS (Pareto-smoothed importance sampling) provides a robust estimate of leave-one-out cross-validation (LOO-CV). LOO-CV refits the model $N$ times, each time leaving out one observation. PSIS approximates this without refitting.

# Demonstrate leave-one-out cross-validation
set.seed(42)
n <- 50
x <- runif(n, 0, 10)
y <- 1 + 2 * x + rnorm(n, sd = 2)

# LOO-CV for linear model

loo_error <- numeric(n)
for (i in 1:n) {
fit <- lm(y[-i] ~ x[-i])
loo_error[i] <- (y[i] - predict(fit, data.frame(x = x[i])))^2
}

cat("Leave-one-out CV (linear model):\n")
cat(" Mean squared prediction error:", round(mean(loo_error), 3), "\n")
cat(" This estimates out-of-sample performance.\n")

Click Run to execute this code in your browser with WebR

6.4.3 Interpreting WAIC differences

The model with the lowest WAIC is preferred. Differences in WAIC between models indicate the relative evidence:

dWAIC < 2: Models are practically equivalent.
2 < dWAIC < 4: Weak evidence for the better model.
dWAIC > 4: Strong evidence for the better model.

Rethinking

Information criteria are estimates, not truths. WAIC and PSIS are estimates of out-of-sample performance. They have uncertainty — the dWAIC has a standard error. When dWAIC is small (less than 2-4), the models are practically indistinguishable. Do not treat the model with the lowest WAIC as "the true model." Instead, use WAIC to identify a set of models that are all reasonable, and then use scientific judgment to choose among them.

Overthinking

The effective number of parameters. The WAIC penalty $p_\text{WAIC}$ is the effective number of parameters, not the actual number. With strong priors, the effective number of parameters is smaller than the actual number because the priors constrain the posterior. This is why WAIC accounts for regularization: a model with many parameters but strong priors may have a smaller $p_\text{WAIC}$ than a model with fewer parameters but flat priors. This is also why WAIC is more general than AIC, which uses the actual number of parameters as the penalty.

Summary of Section 6.4

WAIC = $-2(\text{lppd} - p_\text{WAIC})$ estimates out-of-sample deviance.
lppd measures in-sample fit; $p_\text{WAIC}$ penalizes effective complexity.
PSIS approximates leave-one-out cross-validation without refitting.
dWAIC < 2: models equivalent; dWAIC > 4: strong evidence.
The effective number of parameters accounts for regularization — strong priors reduce $p_\text{WAIC}$ .

Regularization

Using information criteria