2.1 The garden of forking data

Bayesian inference is not a magical procedure. It is a principled way of counting and comparing possibilities. In this section, we build Bayesian inference from the ground up, starting with a simple metaphor: a garden of forking data paths. Each observation we make rules out some paths and leaves others open. The posterior probability of any hypothesis is proportional to the number of paths that remain consistent with both the hypothesis and the data.

This approach avoids the common trap of thinking of Bayesian inference as an opaque formula. Instead, we see it for what it truly is: a systematic way to update our beliefs by eliminating possibilities.

2.1.1 The marble game

Imagine a bag containing four marbles. Each marble is either blue or white. We draw three marbles, one at a time with replacement, and observe the sequence: blue, white, blue. We want to infer the most plausible composition of the bag.

There are five possible compositions (conjectures), from zero blue marbles to four:

| Conjecture | Bag composition | Blue marbles | White marbles | | ---------- | --------------- | ------------ | ------------- | | 0 | ○○○○ | 0 | 4 | | 1 | ●○○○ | 1 | 3 | | 2 | ●●○○ | 2 | 2 | | 3 | ●●●○ | 3 | 1 | | 4 | ●●●● | 4 | 0 |

For each conjecture, we can count the number of ways (paths) to observe the sequence blue–white–blue. The number of paths for each conjecture is the product of the number of ways to draw each marble in the sequence.

For example, under conjecture 2 (two blue, two white):

Ways to draw blue first: $2$
Ways to draw white second: $2$
Ways to draw blue third: $2$
Total paths: $2 \times 2 \times 2 = 8$

# Count the number of ways to observe: blue, white, blue
# for each conjecture about the bag composition
conjectures <- 0:4

# For each conjecture, ways = b _ w _ b (since we draw with replacement)

ways <- sapply(conjectures, function(b) {
w <- 4 - b
b _ w _ b # blue _ white _ blue
})

# Compute relative plausibility (posterior)

posterior <- ways / sum(ways)

data.frame(
conjecture = conjectures,
blue = conjectures,
white = 4 - conjectures,
ways = ways,
posterior = round(posterior, 4)
)

Click Run to execute this code in your browser with WebR

The output shows that conjecture 3 (three blue, one white) has the highest posterior probability, followed closely by conjecture 2. Conjectures 0 and 4 have zero paths — it is impossible to observe both blue and white if the bag contains only one color.

2.1.2 From counts to probabilities

The key insight is that relative plausibility equals relative number of paths. If we assign equal prior probability to each conjecture (the principle of indifference), then the posterior is simply the number of paths divided by the total:

$P(\text{conjecture} \mid \text{data}) = \frac{\text{ways for conjecture}}{\sum_i \text{ways for conjecture}_i}$

This is Bayesian inference in its purest form. No formulas, no integrals — just counting.

# Visualize the posterior distribution
conjectures <- 0:4
ways <- c(0, 3, 8, 9, 0)
posterior <- ways / sum(ways)

barplot(
posterior,
names.arg = conjectures,
xlab = "Conjecture (number of blue marbles)",
ylab = "Posterior probability",
main = "Posterior distribution for marble bag",
col = "steelblue",
border = "darkblue"
)
abline(h = 0, lty = 2, col = "gray40")

Click Run to execute this code in your browser with WebR

The bar plot makes it visually clear that conjecture 3 dominates, but conjecture 2 is not far behind. This uncertainty is the posterior distribution — it does not pick a single "best" answer, but rather tells us how much to believe each possibility.

2.1.3 The globe tossing example

Now let us scale up the same logic. Instead of a bag of marbles, imagine a globe of the Earth. We toss it in the air and catch it. Each time, we note whether our index finger lands on water (W) or land (L). We observe the sequence: W, L, W, W, W, L, W, L, W — that is, 6 waters in 9 tosses.

The parameter of interest is $p$ , the proportion of the globe covered by water. We can approximate $p$ using a grid of possible values and computing, for each value, the number of ways to observe the data.

$P(p \mid D) = \frac{P(D \mid p) \cdot P(p)}{\sum_i P(D \mid p_i) \cdot P(p_i)}$

# Grid approximation for the globe tossing problem
# Define a grid of 20 possible values for p (proportion of water)
p_grid <- seq(from = 0, to = 1, length.out = 20)

# Define a uniform prior: each value of p is equally plausible

prior <- rep(1, 20)

# Compute the likelihood: probability of 6 waters in 9 tosses

# for each value of p on the grid

likelihood <- dbinom(6, size = 9, prob = p_grid)

# Compute the unstandardized posterior

unstd_posterior <- likelihood * prior

# Standardize the posterior so it sums to 1

posterior <- unstd_posterior / sum(unstd_posterior)

# Display the results

data.frame(
p = round(p_grid, 3),
prior = prior,
likelihood = round(likelihood, 6),
posterior = round(posterior, 6)
)

Click Run to execute this code in your browser with WebR

The grid approximation works by:

Defining the parameter grid: We lay out 20 evenly spaced values of $p$ from 0 to 1.
Computing the likelihood: For each $p$ , we use the binomial distribution to compute the probability of observing 6 waters in 9 tosses.
Applying the prior: We multiply the likelihood by the prior. A uniform prior means every value of $p$ starts with the same weight.
Normalizing: We divide by the sum so the posterior is a proper probability distribution.

# Plot the grid-approximate posterior
p_grid <- seq(from = 0, to = 1, length.out = 20)
prior <- rep(1, 20)
likelihood <- dbinom(6, size = 9, prob = p_grid)
posterior <- (likelihood * prior) / sum(likelihood * prior)

plot(
p_grid, posterior,
type = "b",
xlab = "Proportion of water (p)",
ylab = "Posterior probability",
main = "Grid-approximate posterior: 6 waters in 9 tosses",
pch = 16,
col = "steelblue",
lwd = 2
)
grid()

Click Run to execute this code in your browser with WebR

The plot shows a peak near $p \approx 0.67$ (the observed proportion $6/9$ ), with uncertainty spreading across neighboring values. With only 20 grid points, the approximation is coarse, but it captures the essential shape of the posterior.

2.1.4 Sequential updating

Bayesian inference updates naturally as data arrive. Each observation updates the posterior, which becomes the prior for the next observation. This is sequential updating, and it is one of the most elegant features of the Bayesian framework.

Suppose we toss the globe and observe W. Then we toss again and observe L. After each toss, we can update our beliefs:

$P(p \mid W) \propto P(W \mid p) \cdot P(p)$ $P(p \mid W, L) \propto P(L \mid p) \cdot P(p \mid W)$

The order of the data does not matter — the final posterior is the same whether we observe W, L, W or L, W, W. This is because the likelihood factorizes:

$P(D \mid p) = \prod_{i=1}^{n} P(d_i \mid p)$

# Demonstrate sequential updating
# Start with a uniform prior over 100 grid points
p_grid <- seq(0, 1, length.out = 100)
prior <- rep(1, 100)

# Sequence of observations: W, L, W, W, W, L, W, L, W

# W = 1 (water), L = 0 (land)

observations <- c(1, 0, 1, 1, 1, 0, 1, 0, 1)

# Update the posterior after each observation

cat("Sequential updating of the posterior:\n\n")
for (i in seq*along(observations)) {
likelihood <- dbinom(observations[i], size = 1, prob = p_grid)
posterior <- (likelihood * prior) / sum(likelihood _ prior)

# The posterior becomes the prior for the next observation

prior <- posterior
cat(sprintf("After toss %d (%s): peak p = %.3f\n",
i, ifelse(observations[i], "W", "L"),
p_grid[which.max(posterior)]))
}

cat("\nFinal posterior peak:", round(p_grid[which.max(posterior)], 3), "\n")

Click Run to execute this code in your browser with WebR

Notice how the peak of the posterior shifts with each observation. After the first W, the peak moves toward higher $p$ . After the first L, it shifts back slightly. The final peak converges to $p \approx 0.67$ , consistent with 6 waters in 9 tosses.

2.1.5 Evaluating the prior

The choice of prior matters. A uniform prior treats all values of $p$ as equally plausible. But we could also use an informative prior — for example, one that concentrates probability near $p = 0.5$ if we believe the globe has roughly equal parts water and land.

# Compare different priors for the globe tossing problem
p_grid <- seq(0, 1, length.out = 100)
likelihood <- dbinom(6, size = 9, prob = p_grid)

# Prior 1: Uniform

prior_unif <- rep(1, 100)

# Prior 2: Concentrated near 0.5 (step function)

prior_step <- ifelse(p_grid < 0.5, 0, 1)

# Prior 3: Triangular, peaked at 0.5

prior_tri <- 1 - abs(p_grid - 0.5) * 2

# Compute posteriors

post*unif <- (likelihood * prior*unif) / sum(likelihood * prior*unif)
post_step <- (likelihood * prior*step) / sum(likelihood * prior*step)
post_tri <- (likelihood * prior*tri) / sum(likelihood * prior_tri)

# Plot comparison

plot(p_grid, post_unif, type = "l", lwd = 2, col = "steelblue",
xlab = "Proportion of water (p)", ylab = "Posterior density",
main = "Effect of prior on posterior", ylim = c(0, max(post_unif) * 1.1))
lines(p_grid, post_step, lwd = 2, col = "firebrick")
lines(p_grid, post_tri, lwd = 2, col = "darkgreen")
legend("topright",
legend = c("Uniform prior", "Step prior (p >= 0.5)", "Triangular prior"),
col = c("steelblue", "firebrick", "darkgreen"),
lwd = 2, bty = "n")

Click Run to execute this code in your browser with WebR

With only 9 data points, the prior has a noticeable effect on the posterior. As the sample size grows, the likelihood dominates and the prior becomes less influential. This is a fundamental property of Bayesian inference: the data can overwhelm the prior, but the prior always contributes.

Rethinking

The garden metaphor and the principle of indifference. The garden of forking data is more than a teaching device. It encodes a deep truth about inference: every observation eliminates possibilities, and what remains is the posterior. The principle of indifference — assigning equal prior probability to each conjecture — is not a claim about the world, but a statement of ignorance. When we have no reason to prefer one possibility over another, we assign them equal weight. This is the starting point, not the ending point, of inference.

Overthinking

Why the order of data does not matter. The likelihood function for independent and identically distributed (i.i.d.) observations factorizes:

$P(d_1, d_2, \ldots, d_n \mid p) = \prod_{i=1}^{n} P(d_i \mid p)$

Since multiplication is commutative, the order of the factors does not change the product. Therefore, the posterior $P(p \mid d_1, \ldots, d_n)$ is the same regardless of the order in which the data arrive. This is why sequential updating produces the same result as batch updating — the likelihood does not care about the sequence, only the counts.

For the globe tossing example, the likelihood is:

$P(D \mid p) = p^6 (1-p)^3$

This depends only on the number of waters (6) and lands (3), not on their order. The order-independence is a consequence of the i.i.d. assumption. If observations are not independent (for example, if the globe wears down over time), the likelihood would not factorize and the order would matter.

Summary of Section 2.1

Bayesian inference is counting paths through a garden of forking data.
The posterior is proportional to the number of ways the data could arise under each hypothesis, weighted by the prior.
Grid approximation computes the posterior at a finite set of parameter values.
Sequential updating means each posterior becomes the prior for the next observation.
The prior encodes our beliefs before seeing data; with enough data, the likelihood dominates.
The order of observations does not matter under i.i.d. assumptions.

Summary

Building a model