6.3 Regularization

Regularization prevents overfitting by shrinking parameter estimates toward simpler values. In a Bayesian framework, priors play the role of regularization: stronger priors pull estimates toward the prior mean and reduce sensitivity to noise.

6.3.1 Priors as regularization

In Bayesian statistics, the posterior is:

$p(\theta \mid D) \propto p(D \mid \theta) \cdot p(\theta)$

The prior $p(\theta)$ acts as a regularizer: it pulls the posterior toward the prior mean. The stronger the prior, the more regularization. With a flat prior, there is no regularization (equivalent to OLS).

$\hat{\theta}_{\text{Bayes}} = \arg\max_\theta \left\{ \log p(D \mid \theta) + \log p(\theta) \right\}$

# Demonstrate regularization with different priors
set.seed(42)
n <- 20
x <- runif(n, 0, 10)
y <- 1 + 2 * x + rnorm(n, sd = 3)

# No regularization (flat prior = OLS)

fit_ols <- lm(y ~ x)

# Weak regularization (wide prior)

# Simulate posterior with prior: beta ~ Normal(0, 10)

# This barely shrinks the estimate

library(MASS)
post_ols <- mvrnorm(1e4, mu = coef(fit_ols), Sigma = vcov(fit_ols))

# Strong regularization (narrow prior)

# Simulate posterior with prior: beta ~ Normal(0, 0.5)

# This shrinks the slope toward 0

# Approximate: posterior mean = (X'X)^{-1} (X'y + prior_precision * prior_mean)

X <- cbind(1, x)
prior*precision <- diag(c(1/10^2, 1/0.5^2)) # precision = 1/variance
prior_mean <- c(0, 0)
post_mean_strong <- solve(t(X) %*% X + prior*precision) %*% (t(X) %_% y + prior_precision %_% prior*mean)
post_cov_strong <- solve(t(X) %*% X + prior*precision) * summary(fit_ols)$sigma^2

cat("OLS (flat prior):\n")
cat(" intercept:", round(coef(fit_ols)[1], 3), "\n")
cat(" slope:", round(coef(fit_ols)[2], 3), "\n")

cat("\nStrong prior (Normal(0, 0.5) on slope):\n")
cat(" intercept:", round(post_mean_strong[1], 3), "\n")
cat(" slope:", round(post_mean_strong[2], 3), "\n")

cat("\nThe strong prior shrinks the slope toward 0.\n")
cat("This reduces overfitting at the cost of some bias.\n")

Click Run to execute this code in your browser with WebR

6.3.2 Choosing the regularization strength

The strength of regularization depends on the prior's standard deviation:

Wide prior ( $\sigma_\text{prior} = 10$ ): Weak regularization, lets data dominate.
Moderate prior ( $\sigma_\text{prior} = 1$ ): Moderate regularization, balances data and prior.
Narrow prior ( $\sigma_\text{prior} = 0.1$ ): Strong regularization, prior dominates.

# Compare regularization strength
set.seed(42)
n <- 20
x <- runif(n, 0, 10)
y <- 1 + 2 * x + rnorm(n, sd = 3)

X <- cbind(1, x)
sigma2 <- summary(lm(y ~ x))$sigma^2

# Different prior widths for the slope

prior_sds <- c(Inf, 10, 1, 0.5, 0.1)
labels <- c("Flat", "Wide (10)", "Moderate (1)", "Narrow (0.5)", "Very narrow (0.1)")

cat("Slope estimates with different regularization:\n")
for (i in seq*along(prior_sds)) {
if (is.infinite(prior_sds[i])) { # Flat prior = OLS
slope_est <- coef(lm(y ~ x))[2]
} else {
prior_precision <- diag(c(0, 1/prior_sds[i]^2)) # only regularize slope
post_mean <- solve(t(X) %*% X + prior*precision) %*% t(X) %*% y
slope_est <- post_mean[2]
}
cat(sprintf(" %-20s: slope = %.3f\n", labels[i], slope_est))
}

cat("\nStronger regularization shrinks the slope toward 0.\n")

Click Run to execute this code in your browser with WebR

6.3.3 Regularization and prediction

Regularization improves out-of-sample prediction by reducing variance at the cost of some bias. The optimal regularization strength depends on the problem — it is a tuning parameter that can be selected using information criteria.

Rethinking

Regularization is not cheating. Some people think that priors are "subjective" and that regularization is "cheating." But regularization is a principled way to incorporate what we know: that parameters are unlikely to be very large, that simpler models are preferable, and that noise should not be mistaken for signal. The golem does not know this — it will happily fit the noise. Regularization is the modeler's way of telling the golem to be cautious.

Overthinking

Ridge regression and LASSO. The Bayesian model with a normal prior on the coefficients is equivalent to ridge regression in the frequentist literature. The prior $\beta \sim \text{Normal}(0, \tau^2)$ corresponds to the ridge penalty $\lambda \|\beta\|^2$ with $\lambda = \sigma^2 / \tau^2$ . A Laplace prior (double exponential) corresponds to LASSO regression, which can shrink coefficients exactly to zero. These connections show that Bayesian priors and frequentist penalties are two sides of the same coin.

Summary of Section 6.3

Regularization shrinks parameter estimates toward simpler values, reducing overfitting.
In Bayesian models, priors are the regularizers: stronger priors = more shrinkage.
The regularization strength is controlled by the prior's standard deviation.
Regularization improves out-of-sample prediction by reducing variance at the cost of bias.
Normal priors correspond to ridge regression; Laplace priors correspond to LASSO.

Information theory and model performance

Information criteria