A/B test sample size: how to calculate it
The sample size is how many users (like in GA4) each version of an A/B test needs to receive before you can trust the result.Enter the current conversion and the expected conversion in the new version and see the sample instantly.It runs in the browser and does not save the values you type.
Resultado
How did we get to this number?
The closer the new version is to the current conversion, the more users the test requires: half the difference costs, in practice, four times as many users.
To run and monitor growth experiments, get to know the Thesis One.
Example
Example: 5% current conversion, 6% expected conversion, and 1,000 users per day.
Difference to be confirmed: from 5% to 6% (20% more).
Each version: 8,158 users.
Total: 8,158 × 2 = 16,316 users.
Duration: 16,316 ÷ 1,000 = 16.3, that is, about 17 days.
Um caso real: a campanha de 2008
In Barack Obama's presidential campaign in 2008, the team tested the website's registration page: 4 buttons and 6 media, 24 combinations in total.According to Optimizely, which published the case, 310,382 visitors visited the page and each combination was seen by around 13,000 visitors.
The original page converted 8.26% into registrations.The best combination reached 11.6%.
In the calculator, a current conversion of 8.26% and an expected conversion of 11.6% call for 1,258 users per version.They had about 13,000, so there was plenty of traffic to confirm this difference.This is our observation, made later: Optimizely's report does not say whether the campaign calculated the sample before the test.
With 24 combinations, the winner tends to look better than it really is, because the choice is made among many options. This calculator covers two versions.
Fonte: Optimizely, Obama's $60 million dollar experiment.
Why you can't stop testing when one version appears to win
During the test, the difference between the versions goes up and down by pure chance.In the early days, with few people, it tends to be big, and a version can look much better without being so.
Looking at the result several times and stopping when a version seems to win is called peeking. Each look is another chance to catch one of those chance spikes, and the false-positive rate ends up well above the 5% that 95% confidence promises.
A false positive is a winner that doesn't exist: you publish the variation, wait for the improvement and it doesn't appear.
The rule is simple: define the sample before you start, run until atingi-la and only then decide.
Frequently asked questions
What is sample size?
It is the number of users that each version needs to receive for the difference between them to be reliable. With fewer people than that, a real difference can go unnoticed and an apparent difference It could just be chance.
Por que 95% de confidence and 80% power?
They are the most used assumptions in A/B testing. With 95% confidence, if there is no real difference between the versions, the test points to a winner by chance only 5% of the time. With 80% power, if the difference exists in the size you set, the test detects it in 8 out of 10 times.
What is conversion expected in the new version?
It's not a prediction. Think: from which conversion does the change compensate for the effort? The test is sized to confirm this difference safely. If the current conversion is 5% and the minimum that would be worthwhile is 6%, the test was done to confirm the passage from 5% to 6%. If the new version falls below this, the result may be inconclusive. Write down this hypothesis before starting the test. The closer to the current conversion, the more users the test asks.
What counts as a user?
It is the GA4 Users metric. It is not page view: the same user who reloads the page multiple times counts once. Use the same count on current conversion and users per day; if you measure by session, use sessions in both fields.
Posso testar more than two versions?
Yes, but each extra version needs its own sample and increases the chance of a false positive, so plan for more traffic.This calculator considers two versions: the original and a variation.
How long devo rodar um teste?
Until reaching the calculated sample and for at least 7 days, to cover a full week cycle, as the behavior changes between days. If the duration exceeds 60 days, please review the difference you expect to confirm or the page under test.
What to do with little traffic?
Increase the difference you expect to confirm, or try larger changes that are more likely to make a big difference. If you inform users by day, the calculator shows which conversion the test achieves confirm in 30 days.
Quer saber onde o seu growth trava?
The sample size is a number.The Growth Maturity Diagnostic shows what's behind it: acquisition, retention, data and process.It's free and takes just a few minutes.
Make the diagnosisIf the diagnosis points the way, Growth Blueprint I work with you in 4 weekly meetings to turn this into an experimental plan.