Mobile Art Testing Guide

Should I Do a Marketing Art Test?

Early marketing testing allows you to gain insights into how well your game's creative performs during player acquisition. There are two main reasons to run an art test:

• The game’s art is not entirely decided and the team is looking for art direction.
• To determine how marketable an established style is.

It’s important to note that if you aren’t willing to change anything, it’s not worth running a test!

What do I want to learn?

Narrow down your art test to 1 goal. You can run multiple tests if you wish to learn multiple things, but never try to cram a bunch of items into one test or you are likely to invalidate your results. There are several different things you can test; I’ve focused on the main four below.

What theme works best for my art style?

When testing a theme, try to keep everything else in the picture as close as possible. Notice how each image below has 4 characters in a similar grouping. It’s also worth noting that your Click Through Rates (CTR) can be greatly influenced by current pop culture. For example, the release of a new Marvel movie might make people more inclined to click on superhero art. Unfortunately, what’s popular one month might not be so interesting next month.

Which art style works best for my theme?

There are many different styles of art: 3D, pixel, low poly and cartoon, to name a few. The game's art style tells the users a lot about the game. When someone sees Dora the Explorer for the first time, no one needs to tell them it’s a kids' show; they can tell by the art that the show is intended for children. It’s important to think about your audience when considering which art styles to test. Here are a few resources specifically on Art Style if you want to learn more: Creating an Art Style and The Aesthetics of Game Art and Game Design.

Which color palette resonates best with my game?

You can test a color palette, but most of the time it is determined based on the emotions your game is trying to evoke and the intended audience for your game.These guides are pretty interesting if you are trying to figure out your color palette: Color in Video Games: How to Choose a Palette and Picking a Color Palette for Your Game's Artwork.

How marketable is my game art?

This is probably one of the biggest questions we get when it comes to game art. As a general rule, the more casual the game looks the cheaper it is to acquire users for. That said, even within the 3 main categories (casual, mid core & hard core) there is still a huge range! CPI testing is much better than CTR testing to determine marketability.

How will I measure my test?

Is the game built yet? Yes: If there is already a game (even a very basic one) you can get Cost Per Installs (CPI), along with Click Through Rate (CTR) & Conversion Rate (CVR). No: If there is no game you can only get CTR.

CTR Test: CTR lets you gauge general interest in your game.

CPI Test: CPI gives you a deeper read into how marketable a game is. You can gauge general interest (CTR) as well as how likely people are to actually install (CVR) and most importantly, how much it will cost you to acquire users (CPI).

CTR Testing

1. Pick 1 thing you want to test (theme, style, palette, etc.).
2. Create 2-4 creatives that are very different, but focus only on the element you want to test.

• Add 1 creative from another game that you know the CTR for to use as a baseline (see more about baselines below).
• Resource: Facebook Ad Guide

4. Create ads in Facebook using your creatives. Give them the same copy and send them to the same app (make sure to use an app that doesn’t bias one over another).
5. Allow all the ads to get around the same number of impressions and enough to be statistically significant.

The most important part of the test is the range of confidence for your results! The smaller the difference is between art assets, the more users you will need to resolve the test with confidence.

For example: If you run two pieces of art that have CTRs of 1.82% vs. 2.18%, you need over 96,200 impressions to know for sure which is better. Whereas, if their CTRs are 1.70% vs. 2.30%, you only need about 35,000 impressions. Use this simple Statistical Significance Calculator or something similar to find your range.

CPI Testing

1. Pick 1 thing you want to test.
2. Create 2-4 creatives that are very different, but focus only on the element you want to test.

• If possible, run another game at the same time in the same geo as a baseline. CPIs can change a lot month over month; having a comparison point is always helpful.

4. Create ads in Facebook using your creatives. Give them the same copy and send them to your app.
5. Allow all the ads to get around the same number of clicks, enough to be statistically significant (see above).

• It’s important to note that for statistical significance for CPI, your goal is the number of clicks, whereas for CTR it’s the number of impressions.

Why Facebook?

One word: targeting! You can target both age and gender to be sure your art is appealing to your target audience. If you run a test without targeting and get great CTRs on some super-kiddy-looking art, but the game is meant to be for males 18-35, then your test results aren’t very helpful. That said, if your game is intended to be for everyone, then you can test it on any network.

How do I know if the results are good or bad?

Unfortunately, there is no “magic” number we can give you as a comp. The market is constantly changing. That’s why we recommend using a baseline. Your baseline is an image you have used for another game that you can test alongside your new art to help provide perspective on your results.

We finished the test, what now?

1. Make sure your results don’t overlap for statistical significance; if they are too close you may need more data or new creatives with more divergent art.
2. Are the results as good or better than your baseline? If not, you may want to make more changes and re-run the test.

• Note: CTR changes based on art and CVR changes based on your app store page.