A/B Testing: Telling a Real Win from Luck

Your new model beats the old one on every offline test. Does it actually help people? The only way to know is to try it on real users: show some of them the new version, keep the rest on the old one, and compare. That’s an A/B test. Most of an A/B test’s difficulty is telling a real difference from luck, and luck is much bigger than it looks. Here you’ll run thousands of tests in a few seconds and learn the three mistakes that produce most false wins.

Why test on real users at all?#

Offline scores measure a model against data you already have: the clicks, purchases and labels from the past. Real people react to what the model actually shows them, and that can differ in ways no historical dataset captures. A recommender can score higher offline by suggesting things people would have found anyway; a chatbot can get better ratings in testing and longer, more frustrating conversations in use. So teams ship model changes behind an A/B test and let a real outcome decide: purchases, sign-ups, cancellations, time to resolve a ticket.

The recipe: assign each visitor to A (the current version) or B (the new one) at random, so the only systematic difference between the groups is the version they saw. Pick one main number in advance, the primary metric, for example the share of visitors who sign up (the conversion rate). Run it, then compare.

Same page, different results#

With 2,000 visitors per version, the middle 95% of identical tests measured anything from 16% worse to 21% better, and three tests in ten showed a difference of 10% or more. At 200 visitors per version the swings are enormous; at 20,000 they shrink to a few percent. The noise shrinks with the square root of the sample: ten times the visitors, about a third of the wobble.

That’s what the p-value is for. It answers: if A and B were really the same, how often would chance alone produce a difference at least this big? By convention, below 0.05 counts as significant. Run 1,000 A/A tests and about 5% come out significant anyway. Those are false alarms, and 5% is the rate you sign up for when you use 0.05.

How many visitors do you need?#

Now give B a real advantage: it converts 11% of visitors instead of 10%, a 10% improvement. For most products, that would be a result worth having.

At 2,000 visitors per version, only 19% of tests find B’s win. The share of tests that would detect a real effect is called the test’s power, and the usual target is 80%. For 10% versus 11%, that takes about 14,750 visitors per version, and in the simulation 15,000 gets 80%. You can work this number out before the test starts from three things: the conversion rate you have, the smallest improvement worth detecting, and the power you want.

Look at what the small tests claim when they do win: an average improvement of 26%, when the truth is 10%. With a small sample, only the luckiest measurements clear the significance bar, so the winners are systematically exaggerated. Ship on the strength of one and you’ll be promising results that never arrive. An underpowered test usually misses real improvements, and the wins it does report are inflated.

The temptation to peek#

Tests take time, and dashboards update live. It’s natural to check every day and stop as soon as the result looks significant. It feels efficient. It quietly breaks the test.

Checked after every batch, 26.6% of the 1,000 identical tests dipped below 0.05 at some point and would have been stopped as winners. Checked once at the end, 5.6% were significant, about the 5% the test promises. Same data, same tests; only the stopping rule changed.

Two honest fixes: decide the sample size before you start, and don’t act on the result until you reach it. Or use a method designed for continuous monitoring, such as a sequential test, which raises the bar for early stops so the overall false-alarm rate stays at 5%. Many experimentation platforms offer one; check what yours assumes before you peek.

What’s special about testing models#

• Watch guardrail metrics too. Alongside the primary metric, track the things that must not get worse: response time, errors, complaints, cost per request. A bigger model can win on quality and lose on latency.
• Run full weeks. Behaviour on Monday differs from Saturday, and people react to anything new at first, so short tests can catch a novelty effect rather than a lasting one.
• Check the split. If you assigned 50/50 but got 52/48, something in the setup is broken, for example one version crashing for some users. Don’t trust the result until you know why.
• One primary metric, chosen in advance. Look at 20 metrics and one will probably come out significant by chance: the same false-alarm arithmetic as peeking.

In code#

With statsmodels (pip install statsmodels): the sample size before the test, and the significance check after it.

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize, proportions_ztest

# Before the test: users per group to detect 10% -> 11% conversion, 80% of the time, at the usual 5% level
effect = proportion_effectsize(0.11, 0.10)
n = NormalIndPower().solve_power(effect_size=effect, alpha=0.05, power=0.8)
print(f"users per group: {n:,.0f}")     # users per group: 14,744

# After the test: conversions and users in B and in A
stat, p = proportions_ztest(count=[1628, 1480], nobs=[14800, 14800])
print(f"p-value: {p:.3f}")               # p-value: 0.005

statsmodels uses a slightly different approximation from the figure’s formula, hence 14,744 rather than 14,750: close enough either way. And 1,628 against 1,480 sign-ups out of 14,800 each is 11.0% against 10.0%, with p = 0.005: a significant result from a test that was big enough.