Your A/B Test Hit 95% Significance. That's the Worst Time to Stop.
An A/B test should run until it reaches the sample size you calculated before launch, not until the dashboard shows a winner. Stopping at the first sign of 95% significance inflates the real false positive rate from 5% to roughly 26%, according to Evan Miller's simulations. Work out the required sample first, divide by daily traffic, and run at least two full weeks.
That answer annoys people, and I get it. Every testing tool ships a live results view, the chart updates in real time, and somewhere around day four one variant pulls ahead and a little badge says "significant." Everything about the interface is telling you the test is done. The statistics say otherwise, and this article is about the gap between those two things.
The dashboard is built to make you stop early
The technical name for this is peeking. You check a running test, the p-value dips below 0.05, and you call it. The problem is that every check is an implicit hypothesis test, so checking daily for three weeks means you actually ran twenty-one tests and kept the one you liked. Evan Miller worked through the math in How Not To Run an A/B Test back in 2010, and his numbers are still the clearest version I've seen: if you monitor continuously and stop the moment p drops under 0.05, your actual false positive rate is 26.1%, not 5%. Peek ten times over the course of a test and what looks like 1% significance is really 5%.
If that feels abstract, here's the analogy that made it click for me. Stopping at the first significant reading is like flipping a fair coin until you happen to hit a streak of heads, then stopping right there and announcing the coin is rigged. Flip long enough and the streak always arrives. Significance behaves the same way once you let the stopping point float.
And it's not a small distortion. Olga Berezovsky's Data Analysis Journal cites A/A test data, meaning the same page tested against an identical copy of itself, showing 77% of those tests reach statistical significance at some point during the run. Same page. Zero true difference. The dashboard finds a "winner" anyway, three times out of four, if you watch long enough.
If you watch a test until it goes significant, it eventually will. 77% of A/A tests get there with no real difference to find.
The ten-minute math that has to happen before launch
Four inputs decide your sample size, and none of them are negotiable after launch: baseline conversion rate, minimum detectable effect (MDE), significance threshold (usually 95%), and statistical power (usually 80%). Plug them into a calculator like Kameleoon's or CXL's and you get a required number of visitors per variation. Divide that by the page's daily traffic and you have your duration. Ten minutes of work, maybe fifteen if you have to go dig up your baseline rate.
The part that shocks people is how fast the sample requirement grows as the effect shrinks. Say your landing page converts at 3% and you want to detect a 10% relative lift, so 3% moving to 3.3%. A standard calculator puts that around 50,000 visitors per variation, roughly 100,000 total. Want to detect a 5% relative lift instead? You need about four times that. Halving the effect quadruples the sample. This is why "we'll just launch it and see" quietly fails on most sites. Nobody did the division before launch, so nobody realized the test needed four months of traffic before it could say anything trustworthy.
The action here is boring and specific: before you build the variant, run the calculator, divide required sample by daily traffic to that page, and write the end date somewhere the whole team can see it. If the math says 90 or more days, do not shrink the number to fit. Change the test instead. More on that below.
Two full weeks, even when the calculator says five days
Sample size is necessary but not sufficient. A high-traffic page can hit its number in three days, and those three days will lie to you, because Tuesday visitors and Saturday visitors are different people. They arrive from different channels, carry different intent, and convert at different rates. A test that runs Monday through Thursday oversamples work-mode researchers and undersamples weekend browsers, and whichever variant happens to suit the sampled crowd gets flattered.
So the floor is two full weeks, in complete weekly cycles, even when the sample arrives early. Convert's test duration guide goes further: optimizer Kateryna Berestneva's working rule is a minimum of 100 conversions per variation at 95% significance, which in her experience lands most tests in the four to six week range. That roughly matches what I've seen, though I'll admit the 100-conversion floor is more of a field benchmark than a law of nature. It exists because tiny conversion counts make your effect estimate jumpy, and jumpy estimates are how a 40% lift in week one becomes a 6% lift by week five.
Two benchmarks worth writing into the experiment doc: minimum 14 days regardless of traffic, and minimum 100 conversions per variation before you take the significance badge seriously at all.
Low traffic changes what you test, not how the statistics work
If you run the math and your required duration comes back at four months, the statistics are not wrong. The test is. Portent's guidance for low-traffic sites is the honest version of the fix: cut down to two variants so each one gets all the traffic you can give it, and test bigger swings. A button color change needs an enormous sample because its true effect is tiny. A rebuilt offer, a restructured page, a shorter signup flow: those can plausibly move conversion 20% or more, and a 20% MDE needs a small fraction of the traffic a 5% MDE does.
This is also an argument for picking higher-leverage things to test in the first place. The landing page fixes that actually move conversion are mostly structural swings rather than copy tweaks, and that is not an accident. Same with the counterintuitive cases where adding signup friction raised retention. Effects that size are detectable on modest traffic. Word-choice effects mostly are not.
And sometimes the right answer is no test at all. If a page gets 30 conversions a month, ship the change, watch the trend, and accept that you are doing before-and-after observation instead of experimentation. An underpowered A/B test does not give you more rigor than that. It gives you the feeling of rigor, which is worse, because you will act on it.
The honest ways to stop early (there are a few)
None of this means early stopping is always malpractice. There is a whole branch of statistics built for it. Sequential testing designs correct for repeated looks up front: group sequential methods like O'Brien-Fleming spend the error budget so that an early stop only triggers on overwhelming evidence, and always-valid inference methods let you monitor continuously by design. A 2024 paper on setting experiment duration works through the formal approach, and several testing platforms now have sequential engines built in. From what I can tell, they mostly work as advertised.
The distinction that matters: those methods commit to stopping rules before launch. Deciding to stop because today's dashboard looks exciting is a different activity that happens to use the same vocabulary. If your platform offers sequential testing, turn it on and follow its rules. If it does not, then the fixed sample size and the calendar are your rules, and there is no third option that keeps the error rate you think you have.
A prediction, since the incentives here seem lopsided: by the end of 2027 I expect sequential statistics to be the default in every major testing platform, because peeking generates too many false winners for vendors' own case studies to survive it. In the meantime there is a cheap audit you can run this quarter. Rerun your last five declared winners as validation tests. Based on the A/A numbers above, I would expect at least one of the five, probably two, to evaporate on the rerun.
Questions people actually ask while a test is running
Can a test run too long? Yes. Past six to eight weeks, sample pollution creeps in. Cookies get cleared, people switch devices and re-enter the opposite variant, and external events like a sale, a press mention, or an algorithm update land on one part of the calendar. If the math demands more than eight weeks, redesign the test with a bigger minimum detectable effect rather than letting it drift.
What if the test never reaches significance? Then that is the result. It means the true effect, if one exists, is smaller than your MDE, which usually means the change is not worth its maintenance cost. Inconclusive feels like failure and mostly is not. It just killed a change that was not doing anything, which is cheaper than shipping and maintaining it.
Is 90% significance good enough? Depends what a wrong call costs, honestly. Dropping from 95% to 90% doubles your tolerated false positive rate from 5% to 10%. For a reversible, low-stakes change I think that trade is often fine. The rule is the same as everywhere else in this article: pick the threshold before launch, not after you see which threshold your result happens to clear.
Write the stopping rule down before the test starts negotiating
The whole method compresses into one habit. Decide the finish line before launch, in writing: required sample size, minimum 14 days, minimum 100 conversions per variation, significance threshold. Then leave the test alone until it gets there. The same pre-commitment logic applies to which metric you judge it on, by the way. If you have not pinned that down either, start with activation rate rather than raw signups.
The uncomfortable part is that this makes testing slower and less dramatic, and your dashboard will spend weeks showing a "winner" you are not allowed to call yet. I do not have a cure for that itch. What I can tell you is what the replication math says: a good chunk of week-one winners were never winners at all, and the teams with boring, pre-registered stopping rules are the ones whose conversion numbers still mean something a year later.
Notice Me Senpai Editorial
