From Data to Decisions: The Statistical Foundations of Experimentation

The Journey at a Glance

Before we dive in, here’s the full path we’ll walk. Each step builds on the last.

Each coloured block is a section below. By the end, you’ll understand how noisy observations become principled business decisions.

The Setup

Imagine you own a small coffee shop with two baristas — Anna and Ben.

Customers queue up. You start timing how long each one waits from order to drink in hand. Your question is simple:

Is one barista meaningfully faster than the other?

This is the same question behind every A/B test, every feature experiment, every pricing change. The domain differs; the statistical machinery is identical.

Step 1: The Mean — What’s “Typical”?

After timing many customers, you compute each barista’s average:

Mean (Average)

The balancing point of all observations. If you placed every wait time as a weight on a number line, the mean is where the plank balances.

The mean compresses messy reality into a single representative number. It’s the value that minimises the sum of squared distances from every data point — the optimal single summary.

Anna looks 3 seconds faster. But can we trust that number?

Step 2: Standard Deviation — How Consistent?

Two baristas could have the same mean but very different consistency. One is steady, the other is chaos.

Standard Deviation (SD)

How much individual observations spread around the mean. Low SD = consistent. High SD = unpredictable.

Anna’s customers get a tight, predictable experience (most waits between 22–34s). Ben’s range is much wider (19–43s).

Under the hood

Variance is the average squared deviation from the mean:

Standard deviation is the square root of variance — bringing the units back to seconds:

Step 3: The Crucial Shift — From Data to Estimates

Everything so far has described your data. Now we change perspective entirely.

Imagine repeating the same experiment:

• Monday: sample 20 customers → mean = 27.5s
• Tuesday: another 20 → mean = 28.8s
• Wednesday: another 20 → mean = 27.2s
• Thursday: another 20 → mean = 29.1s

Each time you get a slightly different mean. The question becomes:

How much does my estimate wobble from sample to sample?

Think of it like darts:

• SD = how scattered your individual throws are
• SE = how much the centre of your throw pattern shifts between games

Step 4: Standard Error — Quantifying Uncertainty

Standard Error (SE)

The standard deviation of your estimate. It measures how much your calculated mean would vary if you repeated the experiment many times.

The $\sqrt{N}$ is key: more data shrinks your uncertainty.

Step 5: Comparing Anna vs Ben — The A/B Test

Now we move from describing one barista to comparing two. This is the heart of A/B testing.

We define the observed difference:

Anna appears 3 seconds faster. But $\delta$ is built from two noisy estimates — so it has its own uncertainty:

Step 6: The z-Score — Signal vs Noise

We compress the entire comparison into a single number:

z-score

How many standard errors the observed difference is from zero. It’s the ratio of signal (the difference you saw) to noise (the uncertainty in that difference).

The z-score works because of the Central Limit Theorem: averages of many observations are approximately normally distributed, regardless of the underlying data shape. Dividing by SE standardises this to a known scale:

Quick reference:

Step 7: Hypothesis Testing — Making the Call

We now have all the ingredients. Hypothesis testing wraps them into a decision framework.

The p-value answers: “If there were truly no difference, how often would I see a result this extreme just by chance?”

• p < 0.05 → less than 5% chance of seeing this by fluke → reject H₀
• p ≥ 0.05 → the data is consistent with “no real difference” → don’t act yet

In our coffee shop: z = -1.58 gives p ≈ 0.11. Not enough evidence to conclude Anna is faster — we’d need more data.

The Two Types of Error

Every decision framework has failure modes:

Putting It All Together

Here’s the complete pipeline — the full chain from observation to action:

In Code

import numpy as np
from scipy import stats
 
# Observed wait times (seconds)
anna = np.array([25, 30, 28, 29, 27, 26, 31, 24], dtype=float)
ben  = np.array([32, 35, 30, 29, 40, 33, 31, 34], dtype=float)
 
# Step 1-2: Summarise
anna_mean, anna_sd = anna.mean(), anna.std(ddof=1)
ben_mean,  ben_sd  = ben.mean(),  ben.std(ddof=1)
 
# Step 3: Uncertainty
anna_se = anna_sd / np.sqrt(len(anna))
ben_se  = ben_sd  / np.sqrt(len(ben))
 
# Step 4-5: Compare and standardise
delta   = anna_mean - ben_mean
se_delta = np.sqrt(anna_se**2 + ben_se**2)
z       = delta / se_delta
 
# Step 6: Decide
p_value = 2 * stats.norm.sf(abs(z))  # two-tailed test
 
print(f"Anna: mean={anna_mean:.1f}s, SD={anna_sd:.1f}s, SE={anna_se:.2f}s")
print(f"Ben:  mean={ben_mean:.1f}s, SD={ben_sd:.1f}s, SE={ben_se:.2f}s")
print(f"δ = {delta:.1f}s, SE_δ = {se_delta:.2f}s")
print(f"z = {z:.2f}, p = {p_value:.4f}")
print(f"Significant at 5%? {'Yes' if p_value < 0.05 else 'No'}")

What’s Next

This post gives you the statistical foundation — the machinery that powers every experiment.

But knowing the math is only the beginning. In the next post, we’ll tackle the practical side: how to design an A/B test that produces results you can trust, what sample sizes you actually need, and the common pitfalls that invalidate experiments before they even start.

What We Deliberately Left Out

This post focused on building intuition. For further study:

• Confidence intervals — estimating the range of plausible values, not just “significant or not”
• t-tests vs z-tests — when sample sizes are small (< 30), the t-distribution handles extra uncertainty
• Multiple comparisons — testing many hypotheses inflates false positives; Bonferroni and FDR corrections help
• Power analysis — determining sample size before you collect data
• Non-parametric tests — alternatives when normality assumptions don’t hold
• Bayesian approaches — a fundamentally different framework we’ll explore later in this series