1.2 Statistical rethinking

A common misconception is that science proceeds by falsifying hypotheses. But a hypothesis is not a model, and a model is not a hypothesis. Falsifying a model is not the same as falsifying the scientific idea behind it. This section argues for a more humble and nuanced approach to scientific inference, one that treats models as working tools rather than oracles.

The myth of falsification

Karl Popper argued that science advances by falsification: we cannot prove a theory true, but we can prove it false. A theory that survives many attempts at falsification is "corroborated." This is an appealing philosophy, but it has a deep problem when applied to statistical models.

The problem is that we never test a scientific hypothesis directly. We test a model of the hypothesis — a specific statistical procedure with specific assumptions. When the model fails, we cannot tell whether the hypothesis is wrong or whether the model is wrong. The hypothesis and the model are different things.

Consider a simple example. A scientist hypothesizes that " taller people earn more." To test this, she builds a linear regression of income on height. The regression is not significant. Has she falsified the hypothesis?

Not necessarily. The linear regression makes many assumptions:

• The relationship is linear.
• The errors are normally distributed.
• There are no confounding variables.
• The sample is representative.

If any of these assumptions is wrong, the model may fail even if the hypothesis is true. Perhaps the relationship is nonlinear. Perhaps confounders like education or gender distort the association. Perhaps the sample is too small. The model's failure does not falsify the hypothesis — it only falsifies the model.

$\text{model} \neq \text{hypothesis}, \quad \text{falsify model} \neq \text{falsify idea}$

Models as tools, not truths

The lesson is that models are tools, not truths. A model is a provisional device for reasoning about the world. When a model fails, we do not abandon the scientific idea. We repair the model or build a better one.

This is how science actually works. Newtonian mechanics was "falsified" by Einstein's relativity, but we did not abandon the idea that forces cause acceleration. We built a better model. Similarly, when a statistical model fails, we should not abandon the scientific hypothesis. We should ask: what was wrong with the model? Can we fix it?

# Demonstrate: the same hypothesis, two different models
# Hypothesis: taller people earn more
set.seed(42)
n <- 100
height <- rnorm(n, mean = 170, sd = 10)
# True relationship: nonlinear (diminishing returns)
income <- 1000 * sqrt(height - 150) + rnorm(n, sd = 200)

# Model 1: Linear regression (may fail to detect the relationship)

fit_linear <- lm(income ~ height)
p_linear <- summary(fit_linear)$coefficients[2, 4]
cat("Linear model p-value:", round(p_linear, 4), "\n")

# Model 2: Quadratic regression (better fit)

fit_quad <- lm(income ~ height + I(height^2))
p_quad <- summary(fit_quad)$coefficients[2, 4]
cat("Quadratic model p-value:", round(p_quad, 4), "\n")

# Plot both models

plot(height, income, pch = 16, col = "steelblue",
main = "Same hypothesis, different models",
xlab = "Height (cm)", ylab = "Income")
abline(fit_linear, lwd = 2, col = "firebrick")
curve(1000 * sqrt(x - 150), from = 151, to = 190,
add = TRUE, lwd = 2, col = "darkgreen")
legend("topleft",
legend = c("Data", "Linear model", "True relationship"),
col = c("steelblue", "firebrick", "darkgreen"),
pch = c(16, NA, NA), lty = c(NA, 1, 1), lwd = 2)

The linear model may fail to detect the relationship because the true relationship is nonlinear. The quadratic model does better. But neither model tells us whether the hypothesis is true — they only tell us whether the model fits the data. The hypothesis lives at a higher level of abstraction.

The problem with null hypothesis significance testing

Null hypothesis significance testing (NHST) is the dominant statistical procedure in many fields. It works as follows:

1. Define a null hypothesis ($H_0$): usually "no effect" or "no difference."
2. Compute a test statistic from the data.
3. Calculate the $p$-value: the probability of observing a statistic as extreme as ours, assuming $H_0$ is true.
4. If $p < 0.05$, reject $H_0$. Otherwise, fail to reject.

This procedure has several deep problems:

• It does not test the hypothesis of interest. The null hypothesis is usually a straw man. No one believes that the effect is exactly zero. The interesting question is not "is there any effect?" but "how large is the effect?"
• It confuses statistical significance with practical importance. A tiny effect can be statistically significant with a large sample. A large effect can be non-significant with a small sample.
• It encourages binary thinking. "Significant" or "not significant" throws away information. The actual $p$-value, and more importantly the effect size and its uncertainty, are much more informative.
• It is not a measure of evidence. The $p$-value is not the probability that $H_0$ is true. It is the probability of the data given $H_0$, which is a very different thing.

# Demonstrate: statistical significance vs practical importance
# With a large enough sample, any tiny effect becomes "significant"
set.seed(42)

# Tiny true effect: 0.01 standard deviations

n <- 10000
x <- rnorm(n)
y <- 0.01 * x + rnorm(n)

fit <- lm(y ~ x)
p_val <- summary(fit)$coefficients[2, 4]
effect_size <- coef(fit)[2]

cat("Sample size:", n, "\n")
cat("Effect size:", round(effect_size, 4), "(tiny)\n")
cat("p-value:", round(p_val, 6), "\n")
cat("Significant?", ifelse(p_val < 0.05, "YES", "NO"), "\n")
cat("\nWith n = 10000, even a meaningless effect is 'significant'\n")
cat("The golem does not know the effect is trivially small\n")

What should we do instead?

Instead of testing null hypotheses, we should:

1. Estimate effect sizes with their uncertainty. The posterior distribution gives us the full range of plausible values.
2. Compare models rather than testing a single null. Information criteria and cross-validation help us choose the model that predicts best.
3. Use prior information formally. Bayesian inference lets us incorporate what we already know.
4. Simulate and check. Posterior predictive checks let us see whether the model can reproduce the patterns in the data.

This is the approach of this book. It does not abolish hypothesis testing — it replaces it with something better.

Summary of Section 1.2

• A hypothesis is not a model. Falsifying a model does not falsify the scientific idea.
• Models are tools, not truths. When a model fails, we repair or replace it, not abandon the hypothesis.
• NHST has deep problems: it tests straw-man nulls, confuses significance with importance, and encourages binary thinking.
• Instead, we should estimate effect sizes, compare models, use prior information, and simulate to check.
• The Duhem-Quine thesis explains why falsification is insufficient: hypotheses are always tested alongside auxiliary assumptions.