Is Your A/B Test Result Real? Significance, p-values and Effect Size

A green “winner” badge isn’t proof: a trustworthy A/B result needs a small p-value (the difference is unlikely to be chance) and an effect size big enough to matter — from a test you didn’t stop the moment it looked good. Those are three separate questions, and conflating them is how teams ship changes that don’t actually help. Here’s how significance, p-values and effect size fit together, with the A/B Test Significance Calculator, Z-Test Calculator and Effect Size Calculator to run the numbers in your browser.

Step 1: is the difference beyond chance? (the z-test)

An A/B test compares two conversion rates — say 20% for A and 30% for B. The question is whether that 10-point gap reflects a real difference or just the luck of which visitors landed where. The standard answer is a two-proportion z-test: it pools the two rates, computes how many standard errors apart they are (the z statistic), and converts that to a p-value.

The p-value is the probability of a difference at least this large if the two variants were truly identical.

A small p-value (below your chosen threshold α, usually 0.05) means “this data would be surprising if there were no real difference,” so you reject the idea of no difference.

A worked example you can check by hand

Suppose each variant gets 200 visitors. Variant A converts 40 of them (20%) and variant B converts 60 (30%). To run the two-proportion z-test:

1. Pool the rates. Combined conversion is (40 + 60) / (200 + 200) = 100 / 400 = 0.25.
2. Standard error. SE = √[ 0.25 × 0.75 × (1/200 + 1/200) ] = √0.001875 ≈ 0.0433.
3. z statistic. z = (0.30 − 0.20) / 0.0433 ≈ 2.31 — the two rates sit about 2.3 standard errors apart.
4. p-value. A z of 2.31 corresponds to a two-sided p ≈ 0.021, comfortably below the usual α = 0.05.

So this test is statistically significant: a 10-point gap this large would show up only about 2% of the time if the variants were truly identical. Halve the traffic to 100 visitors per side and the same 20%-vs-30% split gives z ≈ 1.63 and p ≈ 0.10 — the identical lift is no longer significant, purely because the sample is smaller. That sensitivity to sample size is exactly why the next two steps matter. The A/B Test Significance Calculator does all four steps for you.

Step 2: what the p-value is not

Two misreadings cause most bad decisions:

• A p-value is not the chance your variant is better. It’s computed assuming no difference. p = 0.02 means “data this extreme is 2% likely if the variants are identical” — not “98% chance B wins.”
• A p-value is not the size of the effect. It blends the effect and the sample size. Which leads straight to the third check.

Step 3: is it big enough to matter? (effect size)

Significance scales with sample size. Feed a million visitors into a test and a 0.1-point lift can be “highly significant” — and commercially pointless. Run 50 visitors and a 10-point lift can miss significance entirely. That’s why significance alone is a trap.

Effect size measures magnitude independent of n. For two means, Cohen’s d expresses the gap in pooled-standard-deviation units (≈0.2 small, 0.5 medium, 0.8 large). For an A/B test, the plain percentage-point lift is the effect that matters — read it next to the p-value, never instead of it.

Cohen’s original benchmarks are rules of thumb, not laws; a “small” effect can be hugely valuable at scale, and a “large” one can be irrelevant if it moves a metric nobody cares about. Use them as a starting sense of magnitude, then judge against your own context:

Cohen’s d	Conventional label	Rough overlap of the two groups
~0.2	small	distributions overlap heavily
~0.5	medium	a difference visible to the naked eye
~0.8	large	clearly separated distributions

The practical habit is to pair every p-value with the effect it came from. “Significant, +0.1 points” and “significant, +10 points” are wildly different business decisions even though both cleared α.

z-test vs t-test: which one?

They answer the same shape of question but under different assumptions:

	z-test	t-test
Population SD	known (or large n)	estimated from the sample
Distribution	normal	Student’s t
Typical use	proportions (A/B tests), large samples	comparing means, smaller samples

Because proportions have a known variance formula, A/B tests use the z-test. Comparing two average order values from modest samples? That’s a t-test.

The cardinal sin: peeking

The fastest way to fake a significant result is to watch the dashboard and stop the moment it turns green. Every extra peek is another chance for random noise to cross the line, and “stop at first p < 0.05” can inflate your false-positive rate several-fold. Decide the sample size before you start, run to it, then evaluate once.

Putting it together

A result you can trust clears all three bars:

1. A z-test p-value below α — the difference is unlikely to be chance.
2. A meaningful effect — the lift is big enough to be worth it.
3. An honest test — pre-set sample size, no peeking.

Because significance and effect size are independent, a finished test lands in one of four places — and only one of them is a clear ship:

	Effect is large	Effect is trivial
p < α (significant)	Ship it — real and worthwhile	Real but likely not worth the cost of the change
p ≥ α (not significant)	Promising — you may be underpowered; gather more data	No evidence of a useful difference

The bottom-right and top-right cells are where teams waste the most effort: shipping changes that are “significant” but move nothing, or chasing a flat result that was never going to pay off. Reading p-value and effect size together keeps you out of both traps.

Run each check locally with the A/B Test Significance Calculator, the Z-Test Calculator and the Effect Size Calculator — your experiment data never leaves your browser.