11.3 Over-dispersed outcomes

Overdispersion occurs when the observed variance exceeds what the nominal distribution allows. Beta-binomial and gamma-Poisson mixtures add a dispersion parameter to absorb this extra variation.

11.3.1 Beta-binomial

The beta-binomial model adds dispersion to the binomial by letting the probability vary:

$p_i \sim \text{Beta}(\bar{p} \theta, (1-\bar{p})\theta)$ $y_i \sim \text{Binomial}(n_i, p_i)$

The variance is:

$\text{Var}(y_i) = n_i \bar{p}_i (1 - \bar{p}_i) \left(1 + \frac{n_i - 1}{\theta + 1}\right)$

When $\theta \to \infty$ , the model reduces to binomial. Smaller $\theta$ means more overdispersion.

# Beta-binomial: simulate and compare with binomial
set.seed(42)
n <- 50
n_trials <- 20

# True probability varies by group (overdispersion)

p_true <- rbeta(n, 2, 3) # varies around mean ~0.4
y <- rbinom(n, n_trials, p_true)

# Binomial model (ignores overdispersion)

m_binom <- glm(cbind(y, n_trials - y) ~ 1, family = binomial)

cat("Binomial model:\n")
cat(" Estimated p:", round(plogis(coef(m*binom)[1]), 3), "\n")
cat(" Mean of y/n:", round(mean(y / n_trials), 3), "\n")
cat(" Var of y/n:", round(var(y / n_trials), 5), "\n")
cat(" Binomial var (np(1-p)):", round(n_trials * mean(y/n*trials) * (1 - mean(y/n_trials)), 3), "\n")

cat("\nOverdispersion:\n")
cat(" Observed variance > binomial variance\n")
cat(" The beta-binomial would absorb this extra variation.\n")

# Plot

hist(y / n_trials, breaks = 10, col = "steelblue",
main = "Beta-binomial: overdispersed proportions",
xlab = "y / n_trials")

Click Run to execute this code in your browser with WebR

11.3.2 Gamma-Poisson (negative binomial)

The gamma-Poisson (equivalent to negative binomial) adds dispersion to the Poisson by letting the rate vary:

$\lambda_i \sim \text{Gamma}\left(\frac{\mu_i}{\phi}, \frac{1}{\phi}\right)$ $y_i \sim \text{Poisson}(\lambda_i)$

The variance is:

$\text{Var}(y_i) = \mu_i + \frac{\mu_i^2}{\phi}$

# Gamma-Poisson (negative binomial): overdispersed counts
set.seed(42)
n <- 100
x <- runif(n, 0, 10)

# Overdispersed: rate varies with Gamma

mu <- exp(0.5 + 0.2 * x)
phi <- 3 # dispersion parameter (smaller = more overdispersion)
lambda <- rgamma(n, shape = mu / phi, rate = 1 / phi)
y <- rpois(n, lambda)

# Poisson (wrong)

m_pois <- glm(y ~ x, family = poisson)

# Negative binomial (correct)

library(MASS)
m_nb <- glm.nb(y ~ x)

cat("Poisson vs Gamma-Poisson (NegBin):\n")
cat(" Poisson: intercept =", round(coef(m_pois)[1], 3),
" slope =", round(coef(m_pois)[2], 3), "\n")
cat(" NegBin: intercept =", round(coef(m_nb)[1], 3),
" slope =", round(coef(m_nb)[2], 3), "\n")
cat(" NegBin theta:", round(m_nb$theta, 2), "(dispersion parameter)\n")

cat("\n AIC comparison:\n")
cat(" Poisson:", round(AIC(m_pois), 1), "\n")
cat(" NegBin: ", round(AIC(m_nb), 1), "\n")

# Check overdispersion

cat("\n Mean:", round(mean(y), 2), " Variance:", round(var(y), 2), "\n")
cat(" Var/Mean ratio:", round(var(y) / mean(y), 2), "(>>1 = overdispersed)\n")

Click Run to execute this code in your browser with WebR

11.3.3 When to use mixture models

Beta-binomial: When binomial data are overdispersed (e.g., clustered binary data, varying probabilities).
Gamma-Poisson: When count data are overdispersed (variance > mean).
Both: When observations are clustered or have unobserved heterogeneity.

Rethinking

Overdispersion is unobserved heterogeneity. When the variance exceeds what the nominal distribution allows, it means there is unobserved variation across observations. Each observation has its own probability or rate, but we do not observe it. The beta-binomial and gamma-Poisson models absorb this variation by adding a distribution over the parameter. This is not a hack — it is a principled way to model heterogeneity. The dispersion parameter tells us how much heterogeneity there is. In Chapter 12, we will see that multilevel models do the same thing, but with observed grouping structure.

Overthinking

Continuous mixtures and the exponential family. The beta-binomial and gamma-Poisson are continuous mixtures — they integrate over a distribution on the parameter. This makes them no longer members of the exponential family, which means they do not have the same nice properties (sufficient statistics, conjugate priors). But they are more realistic. The connection to multilevel models is direct: a multilevel model is a mixture where the mixing distribution is over group-level parameters. The beta-binomial is a multilevel model where each observation is its own group.

Summary of Section 11.3

Beta-binomial: adds dispersion to binomial via a Beta distribution on $p$ .
Gamma-Poisson (negative binomial): adds dispersion to Poisson via a Gamma distribution on $\lambda$ .
Variance exceeds the nominal: $\text{Var}(y) = \mu + \mu^2/\phi$ for gamma-Poisson.
Overdispersion = unobserved heterogeneity — the dispersion parameter measures it.
These are continuous mixtures, connecting directly to multilevel models (Chapter 12).

Zero-inflated outcomes

Summary