The Right Approach to a/b Testing

You’ll need 309,928 visitors per variation. There are 619,856 visitors for an A/B test (the necessary sample size) to detect if the result is statistically valid. However, if you have the same two percent conversion rate and target a 20 percent lift, you’ll need only 19,784 visitors per variation. That’s because it’s statistically easier to match the big winners. If the conversion rate itself is higher than two percent, you’ll also need fewer visitors (four percent conversion rate with a five percent lift = 151,776 visitors per variation). You may be thinking, “Then I’ll try a 20 percent raise” (or, “Why not a 50 to 100 percent or higher raise?”). While 20 percent is realistic, it’s nearly impossible to predict the outcome.

Especially when you don’t have a lot of testing experience and don’t have a clear understanding of what kinds of changes could make a 20 percent difference. What happens when you ignore this and just run the tests? You’ll get a lot of Croatia B2B List inconclusive results, and you won’t be able to tell if your changes made any difference (eg, maybe there’s a 5 percent increase, maybe not, but you need 200,000 more visitors to reach statistical significance). . This is why it is important to calculate the required sample size before each A/B test. There are several sample size calculators available made specifically for A/B testing. My access tool is a/b test: sample size Go ahead and enter your customer’s conversion rate, expected lift (minimum arguable effect).

Always Measure Money

And check if your customer already has enough traffic for A/B testing. Or, if you find this confusing, just Google “A/B Test Sample Size Calculator”. Every popular A/B testing tool has its own. Just a heads up, most of your clients probably have less than 619,856 monthly visitors and this doesn’t mean you can’t A/B test. You can and you must. Just know that you won’t be able to test every little change and you’ll have to be more deliberate with each test. The conversion rate is higher closer to checkout time; and the expected lift (minimum detectable effect) is higher when you’re testing larger changes (more on that later in this article), so you can probably test your cart page or product page. But it’s not just about traffic; you also need a healthy number of purchases.


A lot of people forget this, but if your customer’s store gets less than 100 purchases per month on average, you’re going to have a hard time A/B testing. This is because purchases (transactions) are still your sample size, not just traffic. How many purchases do you need? Sometimes 250 purchases per variation is enough, but more often you will need 500 or more. There is no magic number. It depends on the test, but the more purchases you have, the more confidence you have in the data. If you have a small sample size, your purchases may not be fairly and accurately represented. You may get lucky with some buyers purely by chance, and the result may even appear statistically significant. If you keep the test running longer, there is a high probability that statistical significance will disappear.

A/b Testing Is a Long-term Strategy

Which brings me to my next point… 2. You need a basic understanding of the statistics behind A/B testing a/b test: math This is the part where you can really go crazy… or it can really confuse you. It depends on your relationship with mathematics. This may seem simple, as most A/B testing tools do the complex calculations for you, but the truth is that it is far from simple. If you ignore this part, the risk of getting imaginary results increases significantly. So far, I’ve mentioned terms like sample size (how much traffic you need) and minimum detectable effect (the boost you’re looking for). I also mentioned the term statistical significance (meaning the result is probably not due to chance).

Leave a comment

Your email address will not be published.