The Pitfalls of Running A/B Tests from Ariel Verber

Many people who create digital products have probably heard of the term ‘Designing with Data’. It’s a very obvious practice, that suggests that making intuition-based decisions is not enough, and better decisions are usually supported by quantitative or qualitative evidence.

This leads many teams to run A/B Tests. In short, A/B tests are a way to offer slightly different versions of your product to users of the same initial group, and measure the difference in their behavior. They’re probably one of the best ways to bring actionable data.

The reason A/B tests are so effective, is because they basically mean asking your users absolute questions with 100% truth in the results. For example, by running a simple A/B test you can ask ‘How many extra sales will I make if I offer free shipping worldwide?’. To get an answer for this question, all you need to do is to offer free shipping to 50% of your users, and measure the sales in that group compared to the rest. Then, using simple calculations, you can measure the profitability of adding ‘free shipping’ and decide if it’s worth it or not.

I’ve always been a big advocate of A/B tests, but time led me to learn that they’re highly addictive and sometimes not very justified.

There may be pitfalls that will lead you into making a bad choice. Here are a few examples:
1. Some of the impact may be unforeseen at first
2. Query mistakes are a thing
3. The sample size has to be big enough
4. Numbers don’t have human empathy
5. A/B tests may slow you down

Whole article: https://medium.com/joytunes/the-pitfalls-of-running-a-b-tests-4da7141960d7