6.5 Using information criteria

Model comparison and model averaging use information criteria to decide which models to trust and how to combine their predictions.

6.5.1 Model comparison

Model comparison asks: which model is best? We compare WAIC across models and look at the differences.

# Model comparison with AIC (as a proxy for WAIC)
set.seed(42)
n <- 100
x <- runif(n, 0, 10)
y <- 1 + 2 * x - 0.05 * x^2 + rnorm(n, sd = 2)

# Fit several models

m0 <- lm(y ~ 1) # intercept only
m1 <- lm(y ~ x) # linear
m2 <- lm(y ~ x + I(x^2)) # quadratic
m3 <- lm(y ~ poly(x, 3, raw = TRUE)) # cubic
m4 <- lm(y ~ poly(x, 5, raw = TRUE)) # degree 5

# Compare AIC

aics <- AIC(m0, m1, m2, m3, m4)
daic <- aics$AIC - min(aics$AIC)

cat("Model comparison (AIC):\n")
cat(sprintf("%-20s AIC = %8.2f dAIC = %7.2f\n",
rownames(aics), aics$AIC, daic))

Click Run to execute this code in your browser with WebR

6.5.2 Model averaging

Model averaging asks: how do we combine predictions from multiple models? Rather than choosing a single model, we weight the predictions by the Akaike weights:

$p(\tilde{y} \mid D) = \sum_{m} w_m \, p(\tilde{y} \mid D, m)$

The Akaike weights are:

$w_m = \frac{\exp(-\frac{1}{2} \Delta \text{WAIC}_m)}{\sum_j \exp(-\frac{1}{2} \Delta \text{WAIC}_j)}$

# Model averaging with Akaike weights
set.seed(42)
n <- 100
x <- runif(n, 0, 10)
y <- 1 + 2 * x - 0.05 * x^2 + rnorm(n, sd = 2)

# Fit models

m1 <- lm(y ~ x)
m2 <- lm(y ~ x + I(x^2))
m3 <- lm(y ~ poly(x, 3, raw = TRUE))

# Compute AIC and Akaike weights

aics <- AIC(m1, m2, m3)
daic <- aics$AIC - min(aics$AIC)
weights <- exp(-0.5 _ daic) / sum(exp(-0.5 _ daic))

cat("Akaike weights:\n")
cat(sprintf(" Linear: weight = %.4f\n", weights[1]))
cat(sprintf(" Quadratic: weight = %.4f\n", weights[2]))
cat(sprintf(" Cubic: weight = %.4f\n", weights[3]))

# Averaged prediction at x = 5

x_new <- 5
pred1 <- predict(m1, data.frame(x = x_new))
pred2 <- predict(m2, data.frame(x = x_new))
pred3 <- predict(m3, data.frame(x = x_new))
pred_avg <- sum(weights * c(pred1, pred2, pred3))

cat(sprintf("\nPrediction at x = 5:\n"))
cat(sprintf(" Linear: %.3f\n", pred1))
cat(sprintf(" Quadratic: %.3f\n", pred2))
cat(sprintf(" Cubic: %.3f\n", pred3))
cat(sprintf(" Averaged: %.3f\n", pred_avg))

Click Run to execute this code in your browser with WebR

6.5.3 When to compare vs average

Model comparison: Use when you want to understand which variables matter (causal inference, scientific understanding).
Model averaging: Use when you want the best prediction (forecasting, decision-making).
Both: Use when multiple models are plausible and you want to account for model uncertainty.

Rethinking

Beware the false dichotomy. Model comparison and model averaging are not opposites. They are complementary. Compare models to understand which variables matter and how much evidence supports each model. Average models when you need predictions that account for model uncertainty. The key is to report all plausible models, not just the "best" one. Science progresses by understanding the full landscape of models, not by picking a winner and ignoring the rest.

Summary of Section 6.5

Model comparison uses WAIC differences to rank models.
Model averaging combines predictions weighted by Akaike weights.
Akaike weights: $w_m = \exp(-\frac{1}{2} \Delta \text{WAIC}_m) / \sum_j \exp(-\frac{1}{2} \Delta \text{WAIC}_j)$ .
Use comparison for understanding, averaging for prediction.
Report all plausible models, not just the "best" one.

Information criteria

Summary