Your A/B Test Hit 95% Significance. That's the Worst Time to Stop.
An A/B test should run until it reaches the sample size you calculated before launch, not until the dashboard shows a winner. Stopping at the first sign of 95% significance inflates the real false positive rate from 5% to roughly 26%, according to Evan Miller's simulations. Work