The Experiment That Was Going to Win Anyway

Run an experiment where the two arms are identical. Same page, same code, users split at random, nothing to find.

• Read it once at the planned size: significant 5.0% of the time. Correct.
• Check it ten times and stop when it looks good: 19.3%.
• Read twenty independent metrics instead of one: 64.2%.
• Slice it into twelve segments: 46.4%.

None of those numbers involves an effect. They are what a well-behaved test does when the reading rule is not the one the guarantee was written for. And they are the reason most disagreements about experiment results are really disagreements about decisions that should have been made a month earlier.

First: the test that is too small to be honest

Suppose your conversion rate is 10% and a one point lift would matter. To have an 80% chance of detecting it, you need 14,751 users per arm.

Halving the effect multiplies the traffic by 3.9, because the standard error falls as $1/\sqrt{n}$ and the required sample therefore scales as one over the square of the effect.

Now run it with 2,000 per arm instead. Power drops to 17.8%, so it misses four times out of five. That is expected, and it is not the interesting part.

The interesting part is what it reports on the runs it does catch. Across 20,000 simulated experiments, the average effect among the significant ones is 2.38 times the truth, and 0.8% of them are significant with the wrong sign.

The estimator is not biased. Averaged over all runs it lands on the true effect exactly. What is biased is the average over the runs that passed a filter, and the filter is "the estimate was big enough to clear the threshold" - which at low power requires noise to have helped. This is the winner's curse, and it is why a striking result from a small test so reliably shrinks when someone repeats it properly.

So "we'll treat it as preliminary" is not a defence. An underpowered experiment is not a noisy version of a good one; it is a machine that, whenever it speaks at all, speaks with a badly inflated number and a healthy-looking interval around it.

The figure below computes all of that in closed form rather than by simulation, so the exaggeration moves continuously as you drag the sample size. It opens on the undersized test: 2,000 per arm, 17.8% power, winners overstating the truth by 2.39 times, and 0.8% of them significant backwards. Walk the slider up to 14,751 and two things happen at once. The exaggeration collapses towards one, and the power read-out returns exactly 80.0%, which is the sizing formula and this calculation agreeing on the same threshold.

Second: every extra look is another chance

The estimated difference wanders as data accumulates. The significance threshold is a fixed line. A wandering quantity crosses a fixed line eventually, and a rule that stops at the first crossing keeps the excursion while discarding the return that would have followed.

That is the whole mechanism, and it turns 5.0% into 19.3% at ten looks. Continuous monitoring pushes it towards certainty given enough time.

Two objections that come up and do not hold:

• it is not that early data is noisier and therefore wrong. Each look is a valid test on the data available. The invalid object is the rule combining them.
• it is not fixed by watching the trend instead of the p-value. Any rule that decides to stop based on the result so far has the same structure.

If you need to monitor - and for catching outages you do - use a sequential test or an alpha-spending schedule. Those spend the error budget deliberately. What breaks the guarantee is spending it by accident.

Third: the same arithmetic, sideways

$$P(\text{at least one significant}) = 1 - (1-\alpha)^m$$

for $m$ independent null metrics. Real metrics are usually positively correlated, which brings these rates down somewhat.

At twenty metrics it is more likely than not; at fifty it is nearly guaranteed. Segments behave the same way: twelve of them produce a winner in 46.4% of null experiments, and there is always a story afterwards about why mobile users or new users behave differently. The story is generated after the number, and it is generated just as fluently when the number is noise.

Note the part that surprises people: a larger sample does not fix segments. The rate depends on the number of chances, not their precision. A bigger experiment produces spurious segments just as often, with tighter intervals around them.

Bonferroni brings twenty metrics back to 4.9% by testing each at 0.25%, which is correct and costs a great deal of power. Holm is uniformly better at the same guarantee. Benjamini-Hochberg controls the share of false claims rather than the chance of any, which is the right target when screening candidates and the wrong one when a single false claim is expensive.

But the limitation that matters is not mathematical. A correction only counts the comparisons you declare. The analyst who checked twenty metrics, three segments and four windows before reporting the one that worked made hundreds of implicit comparisons that nothing recorded. The p-value is then a number about a procedure nobody followed.

The cheap fix: buy precision instead of chances

There is a legitimate way to make an experiment cheaper, and it is not any of the above.

Users differ enormously from each other, and most of that has nothing to do with your change. A heavy user was heavy last month too. If you have a pre-experiment measurement $X$ for each user, subtract off the part of the outcome it predicts:

$$Y^{\text{adj}}_i = Y_i - \theta\,(X_i - \bar{X}), \qquad \theta = \frac{\mathrm{Cov}(Y, X)}{\mathrm{Var}(X)}$$

applied identically to both arms. At a pre-period correlation of 0.7 this removes 50.3% of the variance - the theory says $\rho^2 = 0.49$ - so the same precision comes from half the traffic.

It cannot bias the result because $X$ is measured before assignment and cannot have been affected by the treatment. Adjusting for something measured during the experiment breaks exactly that, and can manufacture an effect that is not there.

Two more that cost nothing: trimming a heavy-tailed metric, decided in advance, and stratifying the assignment so the arms are balanced by construction rather than on average.

Neither of this lesson's two numbers needs the simulation that produced it, and the figure below computes both. The adjustment removes exactly rho squared of the variance, so 49% at a correlation of 0.7 and the 50.3% measured above is that figure with 1,500 runs of noise on it. And a clean slice into twelve segments turns up at least one significant result 45.96% of the time, which is what the 46.4% is estimating. Drag either control and watch the trade.

And then: significance is not a decision

The experiment is sized, pre-registered and precise. It returns a significant 0.1% lift. Ship?

Significance answers whether the effect is distinguishable from zero, and with enough traffic almost anything is. The decision needs the interval against the cost:

• 0.02% to 0.18% is real and almost certainly not worth maintaining the feature
• 0.02% to 3.5%, a barely significant 1.8% lift, is a different situation: the experiment has not resolved the question
• a null result with an interval of −0.05% to +0.07% is genuinely informative - you have ruled out anything worth shipping
• a null result with an interval of −4% to +5% has told you nothing at all

That is why the minimum effect worth shipping is fixed at the same moment as the sample size, by the people who will act on it. Fixed in advance it makes the decision arithmetic. Left open it becomes an argument in which everyone reads the same interval in the direction they already preferred.

What the discipline actually is

Write it down before the data arrives:

1. the minimum effect worth acting on, agreed with whoever will act
2. the sample size that detects it, and the decision not to run if the traffic does not exist
3. one primary metric, plus guardrails declared separately
4. a fixed population, duration and stopping rule, with a sequential method if you need to stop early
5. any variance reduction you intend to apply
6. everything else labelled exploration, which produces hypotheses and never conclusions

None of the failures above is a statistical subtlety. Each is the consequence of deciding something after seeing the data that should have been decided before, and the entire discipline of experimentation is the practice of moving those decisions earlier.