A/B Test Calculator

What is statistical significance?

Statistical significance measures the probability that your A/B test results are not due to random chance. A 95% confidence level means there's only a 5% probability the difference between variants occurred by luck. In marketing, 95% confidence (p-value ≤ 0.05) is the standard threshold for declaring a winner. Test headline variations using our Headline Analyzer to optimize emotional impact before running A/B tests.

What is a p-value?

The p-value is the probability that the observed difference happened by random chance. Lower p-values indicate stronger evidence for a real difference:

• p ≤ 0.01: Highly significant (99% confidence) - very strong evidence
• p ≤ 0.05: Significant (95% confidence) - industry standard
• p ≤ 0.10: Marginally significant (90% confidence) - weak evidence
• p > 0.10: Not significant - results could be random noise

How long should I run an A/B test?

Run A/B tests until you achieve both statistical significance AND sufficient sample size:

• Minimum sample size: 350-400 conversions per variant (not visitors)
• Minimum duration: 1-2 full business cycles (7-14 days for most sites)
• Traffic patterns: Capture weekday vs. weekend behavior differences
• Seasonal effects: Avoid starting tests during holidays or promotions
• Early stopping risk: Stopping at first sign of significance leads to 30-50% false positives

Track baseline conversion rates using our Conversion Rate Calculator before launching tests to set realistic improvement targets.

When should I stop an A/B test?

Stop your test when ALL conditions are met:

• Statistical significance achieved: P-value ≤ 0.05 (95% confidence)
• Minimum sample size reached: 350+ conversions per variant
• Full cycle completed: At least 1-2 weeks for complete traffic patterns
• Practical significance: Improvement is large enough to matter (typically 5%+ relative lift)

If no variant reaches significance after 4 weeks with adequate traffic, call it inconclusive. Either the effect is too small to detect or there's no real difference.

Common A/B testing mistakes

Avoid these errors that invalidate results:

• Peeking and stopping early: Checking daily and stopping at first significance inflates false positives by 30-50%
• Too small sample size: Under 350 conversions per variant produces unreliable results
• Unequal traffic split: Use 50/50 splits unless you have specific statistical reasons for unequal allocation
• Testing too many variants: More variants require larger sample sizes (use sequential testing instead)
• Ignoring external factors: Site bugs, seasonal events, or marketing campaigns skew results
• Confusing correlation with causation: Correlation doesn't prove the tested element caused the change

What is relative vs. absolute improvement?

Understanding both metrics prevents misinterpretation:

• Absolute improvement: Percentage point difference (3% → 3.6% = 0.6 percentage points)
• Relative improvement: Percentage change (3% → 3.6% = 20% relative increase)

Example: Control converts at 2%, variant at 2.4%. Absolute improvement is 0.4 percentage points. Relative improvement is 20% (0.4/2.0 × 100). Always report both to avoid misleading claims like "20% improvement" when absolute lift is tiny.