2.5 Summary

Chapter 2 in review

Chapter 2 built Bayesian inference from the ground up, starting with the garden of forking data — the idea that inference is counting paths through a garden of possibilities. Each observation rules out some paths, and the posterior is proportional to the number of paths that remain.

We then learned how to build a model from three ingredients: a likelihood (the data story), parameters (the unknowns), and a prior (pre-data knowledge). The posterior combines all three:

$p(\theta \mid D) \propto p(D \mid \theta) \cdot p(\theta)$

We examined the components of the model in detail: the likelihood is a probability over data (not parameters), the prior is a model of knowledge (not a bias), and parameters are indices of a family of processes (not truths).

Finally, we explored three methods for making the model go: grid approximation (exact but slow), quadratic approximation (fast but approximate), and MCMC (flexible but requires diagnosis).

The small world and the large world

The small world is the self-consistent mathematical world of the model. In the small world, the posterior is well-defined, and the methods we learned can compute it. The large world is the complex reality we hope to understand. The large world does not care about our model — it produces data according to its own processes, which may or may not match our assumptions.

Bayesian inference bridges the small world and the large world by updating the small world in light of data from the large world. But the bridge only works if the small world is a reasonable approximation of the large world. This is why model criticism and comparison are essential — they are the tools for checking whether the small world is adequate.

# Visualize the small world vs the large world
# Small world: our binomial model with p = 0.67
# Large world: the true data-generating process

set.seed(42)

# Small world: model predictions

p <- 0.67
n <- 9
model_predictions <- rbinom(1000, size = n, prob = p)

# Large world: true process (slightly different from our model)

# Maybe the globe is not uniform, or tosses are not independent

true_p <- 0.70
true_data <- rbinom(1000, size = n, prob = true_p)

# Compare

par(mfrow = c(1, 2))
hist(model_predictions, breaks = 0:9 + 0.5, col = rgb(0.27, 0.51, 0.71, 0.5),
main = "Small world: model predictions",
xlab = "Number of waters", ylim = c(0, 300))
hist(true_data, breaks = 0:9 + 0.5, col = rgb(0.8, 0.2, 0.2, 0.5),
main = "Large world: true process",
xlab = "Number of waters", ylim = c(0, 300))
par(mfrow = c(1, 1))

cat("Model mean:", round(mean(model_predictions), 3), "\n")
cat("True mean:", round(mean(true_data), 3), "\n")
cat("The small world is an approximation of the large world\n")

Click Run to execute this code in your browser with WebR

Rethinking

The small world is all we have. We cannot know the large world directly. We can only build small worlds (models) and check them against the large world (data). The goal is not to build a perfect small world — that is impossible — but to build a small world that is useful: one that captures the important features of the large world and is transparent enough to be critiqued and improved. This is the engineering approach to statistics: build, test, critique, and iterate.

Key formulas from Chapter 2

Bayes' theorem: $p(\theta \mid D) = \frac{p(D \mid \theta) \cdot p(\theta)}{p(D)}$
Grid approximation: $p(\theta_i \mid D) = \frac{p(D \mid \theta_i) \cdot p(\theta_i)}{\sum_j p(D \mid \theta_j) \cdot p(\theta_j)}$
Quadratic approximation: $p(\theta \mid D) \approx \text{Normal}(\hat{\theta}, H^{-1})$
Sequential updating: $p(\theta \mid d_1, \ldots, d_n) \propto \prod_{i=1}^n p(d_i \mid \theta) \cdot p(\theta)$
Conjugate prior (binomial): $\text{Beta}(\alpha + w, \beta + n - w)$

Making the model go

Practice