1.4 Summary
Chapter 1 in review
This chapter introduced the golem analogy: statistical models are powerful but mindless tools that do exactly what they are told, but cannot judge whether their instructions are wise. The responsibility for using statistical tools wisely falls on the modeler, not the model.
We saw that falsification is insufficient because we never test hypotheses directly — we test models of hypotheses. When a model fails, we cannot tell whether the hypothesis is wrong or whether the model is wrong. The solution is to treat model building as an iterative engineering process.
We introduced three tools for golem engineering:
- Bayesian inference — a framework for updating beliefs under uncertainty. The posterior distribution gives us the complete answer, with full uncertainty.
-
Multilevel models — a framework for handling structured data through partial pooling. Groups borrow strength from each other, improving estimates for all.
-
Information criteria — a framework for comparing models and estimating out-of-sample predictive accuracy. This prevents overfitting by penalizing complexity.
These three tools form a coherent workflow: specify, fit, critique, compare. The rest of the book develops each tool in depth.
Preview of the book
- Chapters 2-4: Bayesian inference from scratch — grid approximation, quadratic approximation, and the linear model.
- Chapters 5-7: Multivariate models, causal reasoning, and interactions.
- Chapter 8: Markov Chain Monte Carlo — the computational engine for complex models.
- Chapters 9-11: Generalized linear models — binomial, Poisson, ordered, and mixture models.
- Chapters 12-13: Multilevel models and adventures in covariance.
- Chapter 14: Missing data and measurement error.
- Chapter 15: Horoscopes — a reflection on the modeling process.
Every chapter in this book teaches both how to do something and why it matters. The "why" is just as important as the "how", because statistical models are golems — they do what they are told, but they cannot judge whether their instructions are wise. The "why" is what makes you a golem engineer, not just a golem operator.
The book's approach is deliberately constructive rather than destructive. Instead of telling you what not to do (don't use -values, don't use stepwise regression, don't use null hypothesis tests), it teaches you what to do instead. The goal is not to tear down the old golems, but to build better ones.
Practice
The practice problems for Chapter 1 are conceptual rather than computational. They ask you to reflect on the golem analogy, the limitations of falsification, and the three tools of golem engineering.
1E1. What is the difference between a hypothesis and a model? Give an example where falsifying a model does not falsify the hypothesis.
1E2. Why is the -value not the probability that the null hypothesis is true? What is it instead?
1E3. Name the three tools of golem engineering and briefly describe what each one does.
1M1. Suppose you have data on the relationship between temperature and ice cream sales. You fit a linear regression and find a significant positive association. List at least three assumptions of this model that, if violated, could make the conclusion misleading.
1M2. The golem analogy suggests that statistical models cannot judge whether their instructions are wise. Give an example from your own field where a standard statistical procedure might give a "correct" but "unwise" answer.