At Kongregate, all of our published mobile games go through a rigorous test market period (a.k.a. geo-locked launch) lasting at least a month. This is an important phase of the mobile game product cycle, as it allows the team to gauge the strengths and weaknesses of the title through data-driven KPIs, and to make changes to address any issues we observe before we release the game globally, whether they are crashes/bugs, tutorial/retention drop-offs, monetization or ad engagement. In this blog post, we share our learnings from running test market campaigns, with a focus on the pros and cons of running tests in different parts of the world.

Phase 1: The Philippines

The Philippines is a popular country for testing. Players are cheap to acquire with Cost-Per-Installs (CPIs) typically ranging from as low as $0.30 to $1.00. This allows developers/publishers to hit a meaningful number of installs quickly and stay within budget. The caveat is that games with a niche theme and/or art might see higher CPIs; for example, our recently launched Pocket Politics (iOS/Android) saw a much higher CPI because of its American political theme.

For stability and compatibility testing, the Philippines is a great place to consider, as one will observe a wide range of devices and OS versions. Conversely, the in-game KPIs collected in this country are often not reliable or indicative of what KPIs might look like in the U.S. and countries in the Western world. Players are less likely to stay in the game and less likely to engage and make purchases, so don’t despair if you are seeing low early retention and monetization numbers!

Phase 2: Scandinavia (Sweden, Denmark, Finland and Norway)

Once you have verified and improved that stability of the game, you can expand to additional regions. The Nordic countries are great to run tests in because the player behaviors resemble those of U.S. and other Western world countries’ players, while CPIs are usually lower than those of English-speaking countries like Canada, Australia and New Zealand (expect CPIs less than $4).

We often run tests for early retention (D1 and D7), ad engagement, and monetization in these countries, as the KPIs will give us a good idea of what they might look like once a game is launched globally. It is also possible to run A/B tests in this region and get meaningful results. One thing to keep in mind is that since these players are acquired through paid campaigns, KPIs will generally look higher than global metrics. For example, a D1 retention of 50% might turn into 40% globally when you get a mix of organic and paid players, as well as players from both high-performing and low-performing countries.

Phase 3: CANZ (Canada, Australia and New Zealand)

These three countries are the most popular among mobile game developers and publishers because players here most closely resemble those in the U.S. market and thus metrics can be more relied on to estimate U.S. KPIs. As mentioned before, keep in mind that players acquired through UA often show higher KPIs and engagement than those of organics.

The downside to running your test market in these countries is that the costs could be very high, as you will be competing with other developers and publishers who might be willing to bid up the prices to get volume faster.

Practical Things to Keep in Mind

• Timeline
  • The length of the test market period will depend on the metrics you want to test and gain statistical significance on, as well as if any updates are needed to address any under-performing KPIs and/or fix any bugs. For example, to test for D1 and D7 retention, budget your time to get enough players past D7 and make changes to the game. Keep in mind that unless you are flexible with your global launch date, you likely won’t be able to address and test all issues, so it’s important to prioritize!
• Budget
  • To determine what you will be able to test within a given budget, come up with an estimate of CPIs and install counts needed: CPIs vary by region as well as genre (i.e. casual games see lower CPIs and mid/hard-core titles see higher CPIs). A handy calculator to see how many installs you need based on the metrics you want to test can be found here.
• User Acquisition
  • Ad Networks
    • To run UA campaigns to get installs, you will need to set up accounts with ad networks. Examples include Facebook, Unity and Supersonic. Different networks can bring in traffic of various qualities, with players from Facebook Ads usually yielding the highest KPIs.
  • Marketing Creatives
    • Your CPIs will also be affected by the quality of these assets, so it’s important to iterate on these if you are seeing a low CTR and/or CVR. It’s also a good opportunity to test different target audience groups to optimize your campaigns.
  • App Store Optimization
    • Test markets are also a good time to run A/B tests on icons and screenshots, as these will affect conversion rates. Google has built-in functionalities to set up these tests, so be sure to take advantage of them.
• Platform Considerations
  • Approval Times
    • Remember that each time you want to push an update, you will need to wait for platforms to approve your builds.
  • Ratings
    • Monitor crashes each time you are releasing an update. Unexpected crashes/severe bugs that prevent gameplay can quickly lead to low ratings from players. Ratings are important, since they show both the platforms and the players the quality of your game. Average ratings are also harder to lift as the number of ratings grows.

And that’s it for now! If you are interested in how Kongregate can guide you through the test market cycle and help you publish your game, please email bd@kongregate.com for more details! For other ways of testing, and the pros and cons of each, check out our CEO Emily Greer’s presentation at this year’s Game Developers Conference.