The A/B testing sample size formula tells you how many visitors each variant needs before a difference in conversion rate means anything. For a standard two-sided test at 95% confidence and 80% power, you need roughly n = (1.96 + 0.84)² × (p1(1−p1) + p2(1−p2)) / (p2 − p1)² visitors per variant, where p1 is your baseline rate and p2 the rate you hope to detect.
Run that formula against real B2B numbers and the result is uncomfortable. A page converting at 3% needs about 14,000 visitors per variant to reliably detect a 20% relative lift. Plenty of B2B pages do not see that many visitors in a quarter. Below: the formula input by input, computed tables for a range of baseline rates, how long a test takes at a given traffic level, and what to do when the answer is "eight months."
Our Position: Most B2B A/B Tests Are Underpowered
First, the opinion the math supports. Our view is that most B2B A/B tests are stopped long before they have enough data, and that many of the "winners" they produce are noise. The pattern looks like this: a three-week test, a variant ahead by 25%, a green badge, a shipped change, and six months later no visible lift in pipeline.
Underpowered tests miss real effects, which wastes the test slot. Worse, when an underpowered test does cross the significance line, the measured lift tends to be an overestimate, because only the lucky draws make it over the bar. Statisticians call this a magnitude (type M) error. The team ships a change that may do nothing and also books an inflated number into its CRO reporting. Working out the sample size before launch is the cheapest protection.
The A/B Testing Sample Size Formula
The standard formula for comparing two conversion rates (a two-proportion z-test) is:
n per variant = [ zα/2 × √(2 × p̄ × (1 − p̄)) + zβ × √(p1(1 − p1) + p2(1 − p2)) ]² / (p2 − p1)²
Each input has a plain-language meaning:
- p1 (baseline conversion rate): the current conversion rate of the page and goal you are testing. Use the rate for the exact conversion event and the exact page, from the last 4 to 8 weeks, not a site-wide average.
- p2 (target rate): the baseline multiplied by (1 + MDE). With a 3% baseline and a 20% relative MDE, p2 = 3.6%.
- MDE (minimum detectable effect): the smallest relative lift you care about detecting. This is a business decision, not a statistical one. Smaller MDEs need dramatically more traffic, because the difference (p2 − p1) is squared in the denominator. Halve the MDE and you roughly quadruple the sample.
- p̄ (pooled rate): the average of p1 and p2. It describes the noise under the assumption that there is no real difference.
- zα/2 = 1.96: corresponds to a 5% false positive rate (alpha 0.05), split across both tails because the variant could be better or worse.
- zβ = 0.8416: corresponds to 80% power, meaning that if the true lift equals your MDE, the test detects it 80% of the time. The other 20% of the time you miss a real winner.
Many calculators use the simplified unpooled version shown in the first paragraph. At B2B conversion rates the two differ by only a handful of visitors. Our A/B test calculator uses the pooled formula above and also estimates test duration.
Worked Example, Step by Step
Example (illustrative numbers): a demo request page converts at 3%. You want to detect a 20% relative lift.
- Set the rates: p1 = 0.030, p2 = 0.030 × 1.2 = 0.036. The absolute difference is 0.006.
- Pooled rate: p̄ = (0.030 + 0.036) / 2 = 0.033.
- First term: √(2 × 0.033 × 0.967) = √0.06382 = 0.2526. Multiply by 1.96: 0.4951.
- Second term: p1(1−p1) = 0.0291 and p2(1−p2) = 0.0347. √(0.0638) = 0.2526. Multiply by 0.8416: 0.2126.
- Add and square: (0.4951 + 0.2126)² = 0.7077² = 0.5009.
- Divide by the squared difference: 0.5009 / 0.006² = 0.5009 / 0.000036 = 13,914 visitors per variant.
So the test needs about 27,800 visitors in total, and roughly 417 conversions in the control arm alone (13,914 × 3%), before you read the result. As a cross-check, the simplified unpooled formula gives 13,911 per variant, and an arcsine (Cohen's h) approximation gives about 13,890. The three methods agree within 0.2%.
Sample Size Table for B2B Conversion Rates
The table below applies the pooled formula above (alpha 0.05 two-sided, power 0.8) across baseline rates from 1% (for example, demo requests as a share of all visitors to a busy page) to 10% (for example, a micro conversion such as a CTA click). Use your own baseline; these rows are just reference points. Values are visitors per variant, computed in Node.js and rounded up.
| Baseline conversion rate | 10% relative MDE | 20% relative MDE | 30% relative MDE | 50% relative MDE |
|---|---|---|---|---|
| 1% | 163,095 | 42,693 | 19,827 | 7,750 |
| 2% | 80,682 | 21,109 | 9,798 | 3,826 |
| 3% | 53,211 | 13,914 | 6,455 | 2,518 |
| 5% | 31,234 | 8,158 | 3,780 | 1,471 |
| 10% | 14,751 | 3,841 | 1,774 | 686 |
Moving from a 10% MDE to a 50% MDE cuts the required sample by about 20 times. Doubling the baseline rate cuts it roughly in half. You cannot change traffic quickly, but you can choose the size of the change you test and the conversion you measure, and on a small site those are the two levers that matter.
How Long Will the Test Take?
Duration is total required visitors (both variants) divided by the monthly visitors who actually reach the tested page. The table shows months for a 50/50 split at three traffic levels.
| Monthly visitors to the page | Baseline | 10% MDE | 20% MDE | 30% MDE | 50% MDE |
|---|---|---|---|---|---|
| 2,000 | 2% | 80.7 | 21.1 | 9.8 | 3.8 |
| 2,000 | 3% | 53.2 | 13.9 | 6.5 | 2.5 |
| 2,000 | 5% | 31.2 | 8.2 | 3.8 | 1.5 |
| 5,000 | 2% | 32.3 | 8.4 | 3.9 | 1.5 |
| 5,000 | 3% | 21.3 | 5.6 | 2.6 | 1.0 |
| 5,000 | 5% | 12.5 | 3.3 | 1.5 | 0.6 |
| 20,000 | 2% | 8.1 | 2.1 | 1.0 | 0.4 |
| 20,000 | 3% | 5.3 | 1.4 | 0.6 | 0.3 |
| 20,000 | 5% | 3.1 | 0.8 | 0.4 | 0.1 |
Whatever the table says, run every test for at least two full weeks so that weekday and weekend behavior and any campaign spikes are represented in both arms. And treat anything above three months as a test you should probably not run in its current form: page content, campaigns, and traffic mix all drift over that time, and the result describes a site that no longer exists.
At 5,000 monthly visitors and a 3% baseline, a 50% lift is readable in about a month and a 10% lift takes almost two years. Nobody keeps a page unchanged for two years, so the second test will be stopped early and read as if it meant something.
What Most Teams Get Wrong
- Using site-wide traffic in the duration estimate. Only visitors who reach the tested element count. A pricing page test on a 20,000-visitor site may only see 2,000 eligible visitors a month.
- Picking the MDE after the test. If you set the MDE at 30% before launch and the test shows 12%, you do not have a 12% winner. You have an inconclusive test.
- Peeking and stopping on the first green badge. Evan Miller's analysis, How Not To Run an A/B Test, shows that repeated significance testing always increases the false positive rate. In his example (a 50% baseline, a change that does nothing, stopping as soon as the result hits 5% significance or after 150 observations) the false positive rate is 26.1% instead of the nominal 5%.
When the Numbers Say Months: Seven Options
1. Test bigger changes
The cheapest way to cut sample size is to raise the MDE honestly, which means testing changes big enough to plausibly produce it: a new value proposition, a different offer (instant booking versus a contact form), a page tailored to one industry instead of a generic one. Our B2B conversion rate optimisation guide covers how to rank test ideas by expected effect size.
2. Test on higher-traffic pages
Homepage, top blog posts, and high-volume landing pages give you more visitors per month (see our notes on B2B landing page optimization for which pages tend to carry the most decision-stage traffic). A test on a template shared by many pages (all solution pages, all blog posts) pools their traffic into one experiment.
3. Use micro conversions carefully
Micro conversions such as CTA clicks or form starts have higher baselines, so the 10% row of the table applies instead of the 2% row. The catch: a variant can raise clicks and lower qualified demos. Use the micro conversion as the decision metric only when you have evidence it moves with the macro goal, and always report the macro conversion alongside it as a guardrail.
4. Pool segments, then split later
Running separate tests per segment divides your traffic. Run one test across segments, then look at segment-level results as directional only. If you need a segment-specific answer, the test has to be sized for that segment's traffic alone. The 90-day personalization testing roadmap shows how to sequence segment tests so each one gets enough traffic.
5. Use sequential testing
Sequential designs (group sequential tests with pre-planned interim looks, or always-valid methods built into some testing tools) let you check results during the test and stop early when the effect is large, without inflating false positives. The trade-off is a slightly larger maximum sample if the test runs to completion. This is the legitimate version of peeking. If a test would take months, spend the budget on diagnostic CRO tools such as session recordings and form analytics first.
6. Consider a Bayesian approach, with clear expectations
Bayesian A/B testing reports a probability that B beats A and an expected loss if you pick the wrong one. That is easier to act on than a p-value, and it lets you make explicit decisions like "ship if the probability of being better is above 90% and the expected loss is under 0.1 percentage points." For reversible changes, accepting that risk is often rational.
What Bayesian methods do not do is create information. With weak priors, the data needed to be confident about a 20% lift is about the same as in the frequentist table. A 90% "probability to beat" on a tiny sample is still fragile, and checking it daily with a stopping rule still raises the error rate in practice. Bayesian testing changes how you express uncertainty and make decisions. It does not shrink the uncertainty.
7. Do not peek, or plan your peeks
Fix the sample size before launch, write it in the test brief, and do not call the test until you reach it. If stakeholders need interim updates, use a sequential design from the start rather than ad hoc checks. Our experiment plan template has a field for each of these decisions.
A Pre-Launch Checklist
- Pull the baseline rate and monthly eligible visitors for the exact page and conversion (last 4 to 8 weeks).
- Choose the MDE based on the size of the change, and write it down.
- Compute n per variant with the formula or the table above, then the duration.
- If the duration exceeds about three months, redesign the test (bigger change, higher-traffic page, or higher-baseline metric with a guardrail).
- Set the stopping rule: fixed sample, or a sequential plan with pre-planned looks.
For B2B teams, the biggest effect sizes usually come from showing different visitors different content rather than polishing one page for everyone, because a message written for one segment is a bigger change than a new button colour. If you are deciding between a simple split test and a test with several variants or factors, read our comparison of A/B testing vs multivariate testing, which applies the same sample size math to A/B/n and multivariate designs.
When you are ready to run tests on segment-specific experiences, Markettailor's A/B testing for personalized pages lets you split each segment against a control and track the result per segment, so the sample size you planned is the sample size you get.
Frequently Asked Questions
What is the formula for A/B test sample size?
For two conversion rates, n per variant = [1.96 × √(2p̄(1−p̄)) + 0.8416 × √(p1(1−p1) + p2(1−p2))]² / (p2 − p1)², where p1 is the baseline, p2 the target rate, and p̄ their average. This assumes 95% confidence (two-sided) and 80% power.
How many visitors do I need for an A/B test with a 2% conversion rate?
About 21,100 visitors per variant to detect a 20% relative lift, about 9,800 for a 30% lift, and about 3,800 for a 50% lift, at 95% confidence and 80% power. Double these for the total test traffic.
Does Bayesian A/B testing need a smaller sample size?
Not meaningfully, unless you have a strong, justified prior. Bayesian testing gives more intuitive outputs (probability to beat control, expected loss) and supports risk-based decisions, but the amount of data needed to be confident about a given lift is similar to the frequentist calculation.
What minimum detectable effect should a B2B site use?
Pick the MDE from your traffic, not from wishful thinking. Compute what is detectable in six to eight weeks at your page's traffic, then only test changes big enough to plausibly produce that lift. In the duration table above, a page with 5,000 monthly visitors and a 3% baseline can only read lifts of about 30% or more within three months.
Can I stop an A/B test early if it is already significant?
Not with a standard fixed-sample test, because repeated checking inflates false positives. If you need the option to stop early, use a sequential testing design with interim analyses planned before launch.