6.7 Practice

The practice problems for Chapter 6 ask you to compute WAIC for several models, compare them, and perform model averaging.

Practice

Easy problems

6E1. State the three motivating criteria for information criteria.

6E2. Explain the difference between in-sample deviance and out-of-sample deviance.

6E3. When a model overfits, does in-sample deviance increase or decrease? What about out-of-sample deviance?

6E4. What is the effective number of parameters, and how does it differ from the actual number of parameters?

Medium problems

6M1. Write down the WAIC formula and define each term.

6M2. Compute the WAIC for the following models and compare them:

# 6M2: Compare models with AIC (proxy for WAIC)
set.seed(42)
n <- 100
x <- runif(n, 0, 10)
y <- 3 + 1.5 * x + rnorm(n, sd = 2)

m1 <- lm(y ~ x)
m2 <- lm(y ~ x + I(x^2))
m3 <- lm(y ~ 1)

aics <- AIC(m1, m2, m3)
daic <- aics$AIC - min(aics$AIC)
weights <- exp(-0.5 _ daic) / sum(exp(-0.5 _ daic))

cat("Model comparison:\n")
for (i in 1:3) {
cat(sprintf(" %-20s AIC = %7.1f dAIC = %5.1f weight = %.3f\n",
rownames(aics)[i], aics$AIC[i], daic[i], weights[i]))
}

Click Run to execute this code in your browser with WebR

6M3. Simulate data where the true model is quadratic. Fit linear, quadratic, and cubic models. Compare their AIC. Does the quadratic model win?

# 6M3: True model is quadratic
set.seed(42)
n <- 100
x <- runif(n, 0, 10)
y <- 1 + 2 * x - 0.1 * x^2 + rnorm(n, sd = 1)

m1 <- lm(y ~ x)
m2 <- lm(y ~ x + I(x^2))
m3 <- lm(y ~ poly(x, 3, raw = TRUE))

aics <- AIC(m1, m2, m3)
daic <- aics$AIC - min(aics$AIC)

cat("True model is quadratic:\n")
for (i in 1:3) {
cat(sprintf(" %-20s AIC = %7.1f dAIC = %5.1f\n",
rownames(aics)[i], aics$AIC[i], daic[i]))
}
cat("\nThe quadratic model should have the lowest AIC.\n")

Click Run to execute this code in your browser with WebR

Hard problems

6H1. Simulate data with two predictors. Fit models with each predictor alone and both together. Compare WAIC. Which model is preferred?

# 6H1: Model comparison with two predictors
set.seed(42)
n <- 100
x1 <- rnorm(n)
x2 <- 0.5 * x1 + rnorm(n, sd = 0.5)
y <- 1 + 2 * x1 - 1 * x2 + rnorm(n, sd = 1)

# Three models

m_x1 <- lm(y ~ x1)
m_x2 <- lm(y ~ x2)
m_both <- lm(y ~ x1 + x2)

aics <- AIC(m*x1, m_x2, m_both)
daic <- aics$AIC - min(aics$AIC)
weights <- exp(-0.5 * daic) / sum(exp(-0.5 _ daic))

cat("Model comparison:\n")
for (i in 1:3) {
cat(sprintf(" %-20s AIC = %7.1f dAIC = %5.1f weight = %.3f\n",
rownames(aics)[i], aics$AIC[i], daic[i], weights[i]))
}

cat("\nThe model with both predictors should be preferred\n")
cat("because both have real effects.\n")

Click Run to execute this code in your browser with WebR

6H2. Perform model averaging for the models in 6H1. Compute the averaged prediction at $x_1 = 1, x_2 = 0.5$ .

6H3. Demonstrate regularization: fit a model with a strong prior on the slope. Compare the out-of-sample prediction error with and without regularization.

Summary

Building an interaction