4.1 Why normal distributions are normal

The Gaussian (normal) distribution is the natural reference distribution for quantities that are the sum of many small, independent contributions. This section explains why normal distributions appear so often and why they are a good starting point for many models.

4.1.1 The normal distribution

The probability density function of the normal distribution is:

$p(x) = \frac{1}{\sqrt{2\pi}\sigma} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)$

where $\mu$ is the mean and $\sigma$ is the standard deviation. The normal distribution is the most common distribution in statistics, and there are deep reasons for this.

4.1.2 Normal by addition

The central limit theorem (CLT) states that the sum of many independent, identically distributed random variables approaches a normal distribution, regardless of the shape of the individual distributions. This is why normal distributions appear so often in nature: many quantities are the sum of many small effects.

# Demonstrate the central limit theorem
# Sum of 16 uniform random variables
set.seed(42)

# Generate 10,000 sums of 16 uniform(-1, 1) random variables

small_sums <- replicate(1e4, sum(runif(16, -1, 1)))

# Plot the histogram and overlay a normal distribution

hist(small_sums, breaks = 50, freq = FALSE, col = "steelblue", border = "white",
main = "Central limit theorem: sums of uniform variables",
xlab = "Sum of 16 uniform(-1,1) draws")
curve(dnorm(x, mean = mean(small_sums), sd = sd(small_sums)),
add = TRUE, lwd = 2, col = "firebrick")
legend("topright", legend = c("Sums", "Normal approximation"),
col = c("steelblue", "firebrick"), lty = c(NA, 1), lwd = c(NA, 2),
fill = c("steelblue", NA), border = c("white", NA))

cat("Mean of sums:", round(mean(small*sums), 4), "\n")
cat("SD of sums:", round(sd(small_sums), 4), "\n")
cat("Expected mean: 0 (sum of E[uniform(-1,1)] = 0)\n")
cat("Expected SD:", round(sqrt(16 * (1/3)), 4), "(sqrt(16 _ var(uniform(-1,1))))\n")

The histogram of sums is approximately normal, even though the individual variables are uniform. This is the central limit theorem in action. The more variables we sum, the closer the distribution gets to normal.

4.1.3 Normal by multiplication

Normal distributions also arise from multiplication of many small, independent effects. If a quantity is the product of many independent positive factors, its logarithm is the sum of many small effects, which is normal by the CLT. Therefore, the quantity itself is log-normal.

# Normal by multiplication: products of random variables
set.seed(42)

# Product of 12 uniform(0.5, 1.5) variables

# (each factor varies by +/- 50%)

products <- replicate(1e4, prod(runif(12, 0.5, 1.5)))

# The product is approximately log-normal

# log(product) should be approximately normal

log_products <- log(products)

par(mfrow = c(1, 2))
hist(products, breaks = 50, freq = FALSE, col = "steelblue", border = "white",
main = "Products of random variables", xlab = "Product")
hist(log_products, breaks = 50, freq = FALSE, col = "firebrick", border = "white",
main = "Log of products (approximately normal)", xlab = "log(product)")
curve(dnorm(x, mean = mean(log_products), sd = sd(log_products)),
add = TRUE, lwd = 2, col = "darkgreen")
par(mfrow = c(1, 1))

cat("Mean of log(products):", round(mean(log_products), 4), "\n")
cat("SD of log(products):", round(sd(log_products), 4), "\n")

4.1.4 Normal by log-multiplication

The log-normal distribution is common in biology and economics. Many quantities — heights, weights, incomes — are the product of many small factors, and their logarithms are approximately normal.

# Log-normal distribution in nature
# Simulate height as the product of many genetic and environmental factors
set.seed(42)

# Each factor is a small positive deviation from 1

n*factors <- 100
factors <- matrix(runif(n_factors * 1e4, 0.97, 1.03), nrow = 1e4)
heights <- apply(factors, 1, prod) _ 178 # baseline 178 cm

par(mfrow = c(1, 2))
hist(heights, breaks = 50, freq = FALSE, col = "steelblue", border = "white",
main = "Simulated heights (product of factors)", xlab = "Height (cm)")
curve(dnorm(x, mean = mean(heights), sd = sd(heights)),
add = TRUE, lwd = 2, col = "firebrick")

hist(log(heights), breaks = 50, freq = FALSE, col = "firebrick", border = "white",
main = "Log heights (normal)", xlab = "log(height)")
curve(dnorm(x, mean = mean(log(heights)), sd = sd(log(heights))),
add = TRUE, lwd = 2, col = "darkgreen")
par(mfrow = c(1, 1))

cat("Mean height:", round(mean(heights), 2), "cm\n")
cat("SD height:", round(sd(heights), 2), "cm\n")
cat("The normal distribution is a good approximation\n")

4.1.5 Using the normal distribution

The normal distribution is not a law of nature. It is an excellent default when we do not know the exact generating process. The reasons are:

1. Additive processes: Many quantities are sums of small effects → normal by CLT.
2. Multiplicative processes: Many quantities are products of small factors → log-normal, which is close to normal for small variations.
3. Maximum entropy: The normal distribution is the distribution with maximum entropy given a fixed mean and variance. This makes it the least informative distribution consistent with what we know.
4. Mathematical convenience: The normal distribution has nice properties that make computation easier.

Summary of Section 4.1

• The central limit theorem explains why sums of many small effects are normal.
• Multiplication of many small factors produces log-normal distributions, which are close to normal for small variations.
• The normal distribution is the maximum entropy distribution for a given mean and variance.
• It is an excellent default when we do not know the exact generating process.
• It is not a law of nature — always check the model with posterior predictive simulations.