12.2 Varying effects and the underfitting/overfitting trade-off

Varying effects trade off underfitting and overfitting. Complete pooling assumes all groups are identical; no pooling estimates each group independently. Partial pooling is the compromise.

12.2.1 Three pooling strategies

Complete pooling: Ignore group differences. One estimate for all groups. Underfits — misses real group variation.
No pooling: Each group estimated independently. Overfits — noisy estimates for small groups.
Partial pooling: Group estimates drawn from a common distribution. Balances — shrinks toward the mean, with more shrinkage for small groups.

The partial pooling estimate for group $j$ is:

$\hat{\gamma}_j = \frac{\sigma_\gamma^2}{\sigma_\gamma^2 + \sigma_y^2/n_j} \bar{y}_j + \frac{\sigma_y^2/n_j}{\sigma_\gamma^2 + \sigma_y^2/n_j} \bar{y}$

The weight on the group mean $\bar{y}_j$ increases with $n_j$ (more data) and $\sigma_\gamma^2$ (more between-group variation).

# Demonstrate the three pooling strategies
set.seed(42)
n_groups <- 20
group_sizes <- sample(c(5, 10, 25, 50), n_groups, replace = TRUE)

# True group means drawn from a common distribution

true_mu <- 3 # population mean
sigma_between <- 1.5 # between-group SD
sigma_within <- 2 # within-group SD

group_means <- rnorm(n_groups, true_mu, sigma_between)

# Generate data

data_list <- lapply(1:n_groups, function(j) {
rnorm(group_sizes[j], group_means[j], sigma_within)
})

# Compute estimates

no_pool <- sapply(data_list, mean)
complete_pool <- mean(unlist(data_list))

# Partial pooling (manual formula)

sigma*y2 <- sigma_within^2
sigma_g2 <- sigma_between^2
partial_pool <- sapply(1:n_groups, function(j) {
w <- sigma_g2 / (sigma_g2 + sigma_y2 / group_sizes[j])
w * no*pool[j] + (1 - w) * complete_pool
})

cat("Pooling comparison (first 8 groups):\n")
cat(" Group n True No-pool Partial Complete\n")
for (j in 1:8) {
cat(sprintf(" %5d %2d %5.2f %7.2f %7.2f %7.2f\n",
j, group_sizes[j], group_means[j],
no_pool[j], partial_pool[j], complete_pool))
}

# Compute MSE for each strategy

mse_nopool <- mean((no_pool - group_means)^2)
mse_partial <- mean((partial_pool - group_means)^2)
mse_complete <- mean((rep(complete_pool, n_groups) - group_means)^2)

cat("\nMean squared error:\n")
cat(" No pooling: ", round(mse_nopool, 3), "\n")
cat(" Partial pooling:", round(mse_partial, 3), "\n")
cat(" Complete pooling:", round(mse_complete, 3), "\n")

cat("\nPartial pooling has the lowest MSE!\n")

Click Run to execute this code in your browser with WebR

12.2.2 Shrinkage visualization

# Visualize shrinkage
set.seed(42)
n_groups <- 20
group_sizes <- sample(c(5, 10, 25, 50), n_groups, replace = TRUE)
true_mu <- 3
sigma_between <- 1.5
sigma_within <- 2
group_means <- rnorm(n_groups, true_mu, sigma_between)
data_list <- lapply(1:n_groups, function(j) rnorm(group_sizes[j], group_means[j], sigma_within))

no*pool <- sapply(data_list, mean)
complete_pool <- mean(unlist(data_list))
sigma_y2 <- sigma_within^2
sigma_g2 <- sigma_between^2
partial_pool <- sapply(1:n_groups, function(j) {
w <- sigma_g2 / (sigma_g2 + sigma_y2 / group_sizes[j])
w * no*pool[j] + (1 - w) * complete_pool
})

# Plot: shrinkage by group size

plot(group_sizes, no_pool, pch = 16, col = "steelblue",
ylim = range(c(no_pool, partial_pool, group_means)),
xlab = "Group size (n)", ylab = "Estimate",
main = "Shrinkage: no-pooling vs partial pooling")
points(group_sizes, partial_pool, pch = 17, col = "firebrick")
points(group_sizes, group_means, pch = 4, col = "darkgreen", lwd = 2)
abline(h = complete_pool, lty = 2, col = "gray50")
legend("topright", legend = c("No pooling", "Partial pooling", "True mean", "Complete pooling"),
col = c("steelblue", "firebrick", "darkgreen", "gray50"),
pch = c(16, 17, 4, NA), lty = c(NA, NA, NA, 2), lwd = c(NA, NA, 2, 1))

cat("Smaller groups (left) show more shrinkage toward the population mean.\n")
cat("Larger groups (right) are barely shrunk — their data dominates.\n")

Click Run to execute this code in your browser with WebR

Rethinking

Partial pooling predicts better. The reason partial pooling works is not philosophical — it is practical. When predicting for a new group, the best guess is not the group's own data (which may be sparse) nor the grand mean (which ignores group differences). It is a weighted average of the two, where the weights depend on how much data the group has and how much groups vary. This is why partial pooling predicts new groups better than either extreme. It is also why multilevel models are the default for clustered data — they automatically find the right balance.

Summary of Section 12.2

Complete pooling underfits — ignores group differences.
No pooling overfits — noisy estimates for small groups.
Partial pooling balances — shrinks toward the mean, more for small groups.
The shrinkage weight: $\frac{\sigma_\gamma^2}{\sigma_\gamma^2 + \sigma_y^2/n_j}$ — data-driven.
Partial pooling predicts new groups better than either extreme.

Example: Multilevel tadpoles

More than one type of cluster