9.4 Summary

Chapter 9 in review

Chapter 9 connected maximum entropy to generalized linear models. The principle of maximum entropy justifies the choice of likelihood, and the link function extends the linear model to non-Gaussian outcomes.

We covered:

1. Maximum entropy (Section 9.1): The least informative distribution consistent with known constraints. Gaussian for continuous (mean, variance), binomial for binary (expected count), Poisson for counts (expected rate).

2. Generalized linear models (Section 9.2): $g(E[y_i]) = \alpha + \beta x_i$. The link function bridges the linear predictor to the outcome's natural scale. Logit for binary, log for counts, identity for continuous.

3. Maximum entropy priors (Section 9.3): Priors that encode only what we know. Uniform for known range, exponential for known mean, Gaussian for known mean and variance.

Key formulas

• Entropy: $H(p) = -\sum p_i \log p_i$
• GLM: $g(\mathbb{E}[y_i]) = \alpha + \beta x_i$
• Logit link: $\log\frac{p_i}{1-p_i} = \alpha + \beta x_i$
• Log link: $\log(\lambda_i) = \alpha + \beta x_i$
• Logistic function: $p_i = \frac{1}{1 + e^{-(\alpha + \beta x_i)}}$

Key lessons

• The choice of likelihood is not arbitrary — it is the maximum entropy distribution for the outcome type.
• The link function bridges the linear predictor to the outcome's natural scale.
• Priors should encode what we know — maximum entropy is the baseline for ignorance.
• GLMs are a natural generalization of the linear model, not a collection of ad hoc tricks.