How long should an A/B test run?

Until you've collected enough conversions for statistical significance, minimum 2 weeks regardless of sample size. Use a sample size calculator (e.g., AB Testguide) to determine the conversions needed. Most landing page tests need 200–500 conversions per variant.

What statistical significance level should I target?

95% confidence (p < 0.05) is the standard for business decisions. 90% confidence is acceptable for low-cost, easily reversible decisions. Never make decisions based on tests under 90% confidence.

What should I not test?

Don't test elements that haven't been proven to cause problems first. Random 'let's try this' tests waste traffic and time. Use behavioral data (session recordings, heatmaps) to identify specific friction points, then test solutions to those specific problems.

What tools are best for landing page A/B testing?

VWO and Optimizely for enterprise. Google Optimize has been discontinued; consider Convert.com or AB Tasty as alternatives. For simple headline and CTA tests, Crazy Egg's built-in A/B testing is the easiest setup.

Landing Page A/B Testing: What to Test First for Maximum Impact

Q: What statistical significance level should I target?

95% confidence (p < 0.05) is the standard for business decisions. 90% confidence is acceptable for low-cost, easily reversible decisions. Never make decisions based on tests under 90% confidence.

Q: What should I not test?

Don't test elements that haven't been proven to cause problems first. Random 'let's try this' tests waste traffic and time. Use behavioral data (session recordings, heatmaps) to identify specific friction points, then test solutions to those specific problems.

Q: What tools are best for landing page A/B testing?

VWO and Optimizely for enterprise. Google Optimize has been discontinued; consider Convert.com or AB Tasty as alternatives. For simple headline and CTA tests, Crazy Egg's built-in A/B testing is the easiest setup.

Most A/B Tests Fail to Produce Results

The majority of A/B tests run on landing pages produce no statistically significant result. There are two main reasons: the change being tested is too small to detect, or the test doesn't run long enough to reach statistical significance. Both are solvable with better test prioritization and longer run times.

The framework for prioritization: test the highest-impact elements first, estimate the likely effect size, calculate the sample size needed for significance, and only start tests you can run for long enough to conclude. A 0.1% improvement requires 10× the sample size of a 1% improvement.

Priority 1: Test Your Headline

The headline is the single highest-leverage element on any landing page. Changing a headline from generic brand language to specific outcome language can increase conversion rate by 10–40% — an effect size large enough to detect in a few weeks for most products.

Test the entire headline, not word-by-word variations. Compare: 'AI-powered user testing platform' vs 'Find out why users leave your landing page in 28 minutes'. These are meaningfully different hypotheses about what your audience responds to.

Priority 2: Test Your Primary CTA Copy and Position

CTA copy tests are fast to run and can produce significant results. Test generic vs specific: 'Get started' vs 'Test my landing page free'. Test action framing: 'Start free trial' vs 'See my report'. Each tests a different psychological lever.

CTA position matters too: test above-fold CTA vs sticky CTA vs floating CTA. A sticky CTA (fixed to the bottom of the viewport on mobile) typically outperforms a static CTA that requires scrolling to reach.

Priority 3: Test Social Proof Type and Position

Test customer count vs customer logos vs individual testimonials. Test placing social proof immediately below the hero vs in a dedicated section lower on the page. Test specific quantified testimonials vs sentiment testimonials.

The right type of social proof varies by audience. Technical buyers respond to quantified outcomes ('reduced testing time by 85%'). Consumer products respond better to emotional testimonials ('I finally understand why my site wasn't converting'). Run tests to find which resonates with your audience.

Common A/B Testing Mistakes to Avoid

Testing too many things simultaneously: If you change the headline, CTA, and hero image at the same time, you can't know which change caused any difference in conversion rate. Test one element at a time.

Stopping tests too early: A result that looks significant after 50 conversions is often not significant after 500. Use a sample size calculator before starting any test — determine how many conversions per variant you need for 95% confidence, and don't stop until you reach it.