6.1 The problem with parameters

A model with more parameters fits the sample data better but may predict future data worse. Overfitting is the central problem of statistical modeling, and it arises because we optimize the model for the data we have rather than the data we will see.

6.1.1 Overfitting

Overfitting occurs when a model learns the noise in the sample rather than the signal. A model with enough parameters can fit any dataset perfectly, but it will generalize poorly to new data.

$\text{Training deviance} < \text{Test deviance} \quad \text{(overfitting)}$

# Demonstrate overfitting
# More parameters always improve training fit but may hurt test fit

set.seed(42)
n <- 20

# Generate simple data: y = 1 + 2x + noise

x <- runif(n, 0, 10)
y <- 1 + 2 * x + rnorm(n, sd = 2)

# Split into training and test

train_idx <- 1:15
test_idx <- 16:20

x_train <- x[train_idx]
y_train <- y[train_idx]
x_test <- x[test_idx]
y_test <- y[test_idx]

# Fit models of increasing complexity

degrees <- 1:6
train_error <- numeric(length(degrees))
test_error <- numeric(length(degrees))

for (i in seq_along(degrees)) {
fit <- lm(y_train ~ poly(x_train, degrees[i], raw = TRUE))
train_error[i] <- mean((y_train - predict(fit))^2)
test_error[i] <- mean((y_test - predict(fit, data.frame(x_train = x_test)))^2)
}

# Plot

plot(degrees, train_error, type = "b", pch = 16, col = "steelblue", lwd = 2,
xlab = "Polynomial degree", ylab = "Mean squared error",
main = "Overfitting: training vs test error", ylim = c(0, max(test_error) * 1.1))
lines(degrees, test_error, type = "b", pch = 17, col = "firebrick", lwd = 2)
legend("topleft", legend = c("Training error", "Test error"),
col = c("steelblue", "firebrick"), pch = c(16, 17), lwd = 2)

cat("Training error decreases with complexity\n")
cat("Test error decreases then increases (overfitting)\n")

Click Run to execute this code in your browser with WebR

6.1.2 Underfitting

Underfitting is the opposite problem: the model is too simple to capture the signal. An underfit model has high bias but low variance. The goal is to find the sweet spot — a model that captures the signal without learning the noise.

# Underfitting vs overfitting
set.seed(42)
n <- 50
x <- runif(n, 0, 10)
# True relationship: quadratic
y <- 1 + 2 * x - 0.1 * x^2 + rnorm(n, sd = 1)

# Underfit: linear model

m1 <- lm(y ~ x)

# Just right: quadratic model

m2 <- lm(y ~ x + I(x^2))

# Overfit: degree 10

m10 <- lm(y ~ poly(x, 10, raw = TRUE))

cat("Underfit (linear): R² =", round(summary(m1)$r.squared, 4), "\n")
cat("Just right (quadratic): R² =", round(summary(m2)$r.squared, 4), "\n")
cat("Overfit (degree 10): R² =", round(summary(m10)$r.squared, 4), "\n")

# Plot

x_seq <- seq(0, 10, length.out = 200)
par(mfrow = c(1, 3))
for (model_name in c("m1", "m2", "m10")) {
fit <- get(model_name)
plot(x, y, pch = 16, col = "steelblue",
main = model_name, xlab = "x", ylab = "y", ylim = range(y))
lines(x_seq, predict(fit, data.frame(x = x_seq)), lwd = 2, col = "firebrick")
}
par(mfrow = c(1, 1))

Click Run to execute this code in your browser with WebR

6.1.3 The bias-variance tradeoff

Every model faces a bias-variance tradeoff:

Bias: Error from wrong assumptions (underfitting). Simple models have high bias.
Variance: Error from sensitivity to noise (overfitting). Complex models have high variance.
Total error = Bias² + Variance + Irreducible error

The goal is to minimize total error, not just bias or variance.

Rethinking

The goal is prediction, not fit. The most common mistake in statistical modeling is optimizing for fit rather than prediction. A model that fits the sample perfectly is not a good model — it is an overfit model. The goal is to predict new data, and for that, we need models that capture the signal without the noise. This is why information criteria (Section 6.4) estimate out-of-sample performance, not in-sample fit.

Overthinking

The bias-variance decomposition. For a model $\hat{f}(x)$ predicting $y = f(x) + \epsilon$ with $E[\epsilon] = 0$ and $\text{Var}(\epsilon) = \sigma^2$ :

$E[(y - \hat{f}(x))^2] = \underbrace{(E[\hat{f}(x)] - f(x))^2}_{\text{Bias}^2} + \underbrace{\text{Var}(\hat{f}(x))}_{\text{Variance}} + \underbrace{\sigma^2}_{\text{Irreducible}}$

As model complexity increases, bias decreases (the model can represent the true $f$ ) but variance increases (the model is more sensitive to the particular sample). The optimal complexity minimizes the sum of bias² and variance.

Summary of Section 6.1

Overfitting: More parameters improve training fit but hurt test prediction.
Underfitting: Too few parameters miss the signal.
The bias-variance tradeoff governs model complexity: bias decreases but variance increases with complexity.
The goal is prediction, not fit — optimize for out-of-sample performance.

Practice

Information theory and model performance