An A/B test splits traffic between two variants — A and B — to measure which drives a better outcome. The result is only trustworthy once each variant has accumulated a large enough sample size to distinguish a real difference from random noise at a chosen confidence level.
Why it matters for agencies
The most common testing mistake is calling a winner too early, on too little data. Required sample size grows sharply as the effect you want to detect shrinks, so small lifts need lots of traffic to prove. Sizing the test up front sets expectations on how long it must run and prevents shipping changes based on chance.
Run the numbers with the free A/B test sample size calculator.