5.3 When adding variables hurts

Adding variables does not always improve a model. Two common problems are multicollinearity (predictors are too correlated) and post-treatment bias (controlling for a consequence of the treatment).

5.3.1 Multicollinearity

Multicollinearity occurs when two or more predictors are highly correlated. When predictors are nearly identical, the model cannot distinguish their effects, and the standard errors of the coefficients become very large.

$\text{SE}(\hat{\beta}) \nearrow \quad \text{as} \quad \text{cor}(X_1, X_2) \nearrow 1$

# Demonstrate multicollinearity
set.seed(42)
n <- 100

# Two highly correlated predictors

x1 <- rnorm(n)
x2 <- x1 + rnorm(n, sd = 0.1) # very correlated with x1

# Outcome depends on x1 only

y <- 1 + 2 * x1 + rnorm(n, sd = 1)

# Fit regression with both predictors

fit_multi <- lm(y ~ x1 + x2)
cat("Multivariate model (x1 and x2 correlated):\n")
cat(" beta_x1:", round(coef(fit_multi)[2], 3),
" SE:", round(summary(fit_multi)$coefficients[2, 2], 3), "\n")
cat("  beta_x2:", round(coef(fit_multi)[3], 3),
  "  SE:", round(summary(fit_multi)$coefficients[3, 2], 3), "\n")

# Compare with model using only x1

fit_single <- lm(y ~ x1)
cat("\nSingle predictor model (x1 only):\n")
cat(" beta_x1:", round(coef(fit_single)[2], 3),
" SE:", round(summary(fit_single)$coefficients[2, 2], 3), "\n")

cat("\nCorrelation between x1 and x2:", round(cor(x1, x2), 3), "\n")
cat("Multicollinearity inflates the standard errors.\n")

Click Run to execute this code in your browser with WebR

The standard errors are much larger in the multivariate model because the model cannot tell which predictor is responsible for the effect. The coefficients are also unstable — small changes in the data can cause large changes in the estimates.

5.3.2 Post-treatment bias

Post-treatment bias occurs when we control for a variable that is a consequence of the treatment. This blocks the causal path we want to estimate.

# Demonstrate post-treatment bias
set.seed(42)
n <- 200

# Treatment: tutoring program

tutoring <- rbinom(n, 1, 0.5)

# Post-treatment variable: study hours (influenced by tutoring)

study_hours <- 2 + 1.5 * tutoring + rnorm(n, sd = 1)

# Outcome: test score (influenced by both tutoring and study hours)

# Total effect of tutoring = direct + indirect through study hours

score <- 50 + 3 _ tutoring + 2 _ study_hours + rnorm(n, sd = 3)

# Correct model: estimate total effect of tutoring

m_total <- lm(score ~ tutoring)
cat("Total effect model (score ~ tutoring):\n")
cat(" beta_tutoring:", round(coef(m_total)[2], 3), "\n")

# Post-treatment bias: controlling for study hours

m_bias <- lm(score ~ tutoring + study_hours)
cat("\nPost-treatment model (score ~ tutoring + study_hours):\n")
cat(" beta_tutoring:", round(coef(m_bias)[2], 3), "\n")
cat(" beta_study:", round(coef(m_bias)[3], 3), "\n")

cat("\nThe total effect (6.0) is split into direct (3.0) and indirect (3.0)\n")
cat("Controlling for study_hours blocks the indirect path\n")
cat("and underestimates the total effect of tutoring.\n")

Click Run to execute this code in your browser with WebR

The total effect of tutoring includes both the direct effect and the indirect effect through study hours. By conditioning on study hours, we block the indirect path and underestimate the total effect.

5.3.3 When to add variables

The decision to add a variable should be guided by the causal graph, not by statistical criteria:

Add variables that are common causes (confounders) of the predictor and outcome.
Do not add variables that are consequences of the predictor (post-treatment variables).
Be careful with variables that are highly correlated with other predictors (multicollinearity).
Add variables that reveal masked relationships (indirect paths).

Rethinking

The causal graph is the strategy. Statistical models are tools, but tools need a strategy. The causal graph (DAG) is the strategy. It tells you which variables to include, which to exclude, and which to condition on. Without a causal graph, you are just turning knobs and hoping for the best. With a causal graph, you can make principled decisions about model specification. This is the approach of structural causal models, and it is the foundation of modern causal inference.

Overthinking

The variance inflation factor. The variance inflation factor (VIF) measures how much multicollinearity inflates the variance of a coefficient:

$\text{VIF}_j = \frac{1}{1 - R_j^2}$

where $R_j^2$ is the $R^2$ from regressing predictor $j$ on all other predictors. A VIF of 1 means no multicollinearity. A VIF of 10 means the variance is 10 times larger than it would be without multicollinearity. As a rule of thumb, VIF > 5 indicates problematic multicollinearity. However, multicollinearity is not always a problem — if you only care about prediction, multicollinearity does not hurt. It only hurts when you want to interpret individual coefficients.

Summary of Section 5.3

Multicollinearity: Highly correlated predictors inflate standard errors and make coefficients unstable.
Post-treatment bias: Controlling for a consequence of the treatment blocks the causal path and biases the total effect.
The decision to add variables should be guided by the causal graph, not by statistical criteria.
Confounders should be added; post-treatment variables should not.
The VIF measures the severity of multicollinearity.

Masked relationship

Categorical variables