A/B testing is a randomized controlled experiment in which visitors are split at random between two versions of a page, email or feature, the control (A) and a variant (B), and a predefined metric is compared to see whether the difference between them is larger than chance would explain. It is the standard way to learn whether a change caused an improvement rather than coincided with one.
The method is well established in online products; the practical guide by Kohavi and colleagues remains a standard reference (Kohavi et al., "Controlled experiments on the web", Data Mining and Knowledge Discovery, 2009).
How A/B testing works
- Hypothesis and metric. State the change, the expected effect and one primary metric, such as demo request rate.
- Randomization. Each visitor is assigned to A or B at random and kept there on return visits, so the groups differ only by the change.
- Sample size. Decide in advance how many visitors each version needs, based on the baseline rate, the smallest lift worth detecting (minimum detectable effect), the significance level and the power.
- Run. Let the test reach that sample, ideally across whole weeks. Do not stop early because one side looks ahead.
- Analyze. Compare the rates. A result is statistically significant at 5% when, if there were truly no difference, a gap this large would appear less than 5% of the time.
Two settings matter. Significance level (usually 5%, often called 95% confidence) controls false positives. Power (usually 80%) is the chance of detecting the effect if it is real; an under-powered test mostly produces "no result".
Example
Example: a demo page converts at 3%. You want to detect a 20% relative lift (3.0% to 3.6%) with a two-sided test at 5% significance and 80% power. The standard two-proportion formula is:
n per variant = [zα/2 x √(2 p̄(1 − p̄)) + zβ x √(p₁(1 − p₁) + p₂(1 − p₂))]² / (p₂ − p₁)²
- p₁ = 0.030, p₂ = 0.036, p̄ = 0.033, zα/2 = 1.96, zβ = 0.84
- First term: 1.96 x √(2 x 0.033 x 0.967) = 1.96 x 0.2526 = 0.495
- Second term: 0.84 x √(0.0291 + 0.0347) = 0.84 x 0.2526 = 0.213
- Numerator: (0.495 + 0.213)² = 0.501; denominator: 0.006² = 0.000036
- n ≈ 13,900 visitors per variant, about 27,800 in total
At 2,000 visitors a week to that page, the test needs roughly 14 weeks. That is the useful output: before building anything, you know this page cannot support a test for a 20% lift in a reasonable time, so you test a bolder change, use a higher-traffic page, or ship and monitor. You can reproduce the number with our A/B test calculator or Evan Miller's sample size calculator.
A/B vs A/B/n vs multivariate testing
| A/B | A/B/n | Multivariate (MVT) | |
|---|---|---|---|
| What varies | One control vs one variant | One control vs several variants | Several elements at once, every combination |
| Traffic needed | Lowest | Grows with each variant | Highest: traffic splits across all combinations |
| Answers | Is B better than A? | Which of several versions is best? | Which combination of elements works, and do they interact? |
| Fits | Most B2B pages | Headlines or offers with real traffic | High-traffic pages only |
Nielsen Norman Group summarizes the trade-off: multivariate tests suit incremental refinements on high-traffic pages, while A/B tests can compare radically different designs (NN/g, "Multivariate vs. A/B Testing").
How B2B teams use A/B testing
- Testing demo and contact form length and wording.
- Comparing value propositions and headlines on high-traffic landing pages.
- Validating personalization: segment-specific content vs the generic page, with a holdout.
- Pricing page layout and plan presentation.
- Email subject lines and nurture content, where volume allows.
Frequently Asked Questions
How long should an A/B test run?
Until it reaches the sample size calculated before launch, and for at least one full week, preferably two, so all weekdays are represented. Divide the required visitors per variant by the weekly traffic per variant to estimate the duration. If the answer is several months, the test is not viable as designed.
What is a good minimum detectable effect?
The smallest lift that would be worth acting on, given the cost of the change. Low-traffic B2B pages usually need a larger minimum detectable effect, often 20% relative or more, to finish in reasonable time. Smaller effects can be real but need far more traffic, because sample size grows roughly with the inverse square of the effect.
Can you A/B test with low traffic?
Yes, but test bigger changes, use the highest-traffic step in the funnel, and pick a metric that happens often, such as clicks to the demo page rather than closed deals. When even that is too slow, make the change based on research and monitor before-and-after, accepting that the evidence is weaker.
Related
Plan tests with the A/B test plan template, read the deeper walkthrough of A/B test sample size, and choose a test design with A/B testing vs multivariate testing. For the bigger process, see conversion rate optimization.