Don’t Rush Your A/B Test!
If you have been working in game development for a while I have no doubt that you’ve used A/B testing at least once. It’s a powerful tool, allowing you to check various hypotheses with data before introducing any changes to the entire game’s user base. There is no space for gut-feeling here; we should rely only on pure numbers. Sounds pretty straightforward -- could anything go wrong? Apparently, it could. In the world of A/B testing there are a lot of pitfalls. Here is a list of the most common ones: Running the A/B test without a clear hypothesis and metric to measure Jumping to conclusions before getting statistical significance Uneven distribution in the variants between geos, platforms and whatnot Incorrect implementation of the A/B test conditions Issues with the correct assignment of an A/B test group Too short of an A/B test (i.e. one day only), resulting in biased data In this article we will be focusing on the issues caused by rushed decisions. Here at Kongregate, we care