6.6 Summary

Chapter 6 in review

Chapter 6 introduced the information-theoretic foundation of model comparison. Overfitting is the central disease, regularization is the cure, and information criteria are the diagnostic.

We covered:

The problem with parameters (Section 6.1): Overfitting occurs when models learn noise. The bias-variance tradeoff governs model complexity.
Information theory (Section 6.2): Entropy, KL divergence, and expected log-likelihood provide the language for measuring model performance.
Regularization (Section 6.3): Priors act as regularizers, shrinking estimates toward simpler values and improving out-of-sample prediction.
Information criteria (Section 6.4): WAIC and PSIS estimate out-of-sample performance without leaving the sample. WAIC penalizes the effective number of parameters.
Using information criteria (Section 6.5): Model comparison ranks models; model averaging combines predictions. Both should be used to report the full landscape of plausible models.

Key formulas

Entropy: $H(p) = -\sum p_i \log p_i$
KL divergence: $D_{\text{KL}}(p \| q) = \sum p_i \log(p_i / q_i)$
WAIC: $\text{WAIC} = -2(\text{lppd} - p_\text{WAIC})$
Akaike weights: $w_m = \exp(-\frac{1}{2} \Delta \text{WAIC}_m) / \sum_j \exp(-\frac{1}{2} \Delta \text{WAIC}_j)$

Rethinking

Prediction is the ultimate test. Chapter 6 shows that the goal of modeling is not to fit the data but to predict new data. Overfitting is the natural consequence of optimizing for fit. Regularization and information criteria help us balance fit and complexity, producing models that generalize. This is the practical foundation for everything that follows — from interactions (Chapter 7) to multilevel models (Chapter 12), the question is always: does the model predict well out-of-sample?

Using information criteria

Practice