A Beginner's Guide to User Testing

A few years ago, I was baffled by the performance of an email my team was sending to our users. At a glance, the unique open and click-through rates were fantastic by our industry standards, but when the users landed in our app, the bounce rate was uncomfortably high. Paradoxical, indeed. When we looked at the feature activity data, it seemed that users were either bailing seconds after arriving or clicking aimlessly before eventually leaving, missing the feature we told them about entirely. It was almost as if they had no idea what to do once they got there.

Determined to understand why the email was clearly failing at what we wanted it to do (and under a somewhat intimidating time constraint of the upcoming sprint), I printed it out and took it to the food carts down the street. I had every food cart owner read the email, and in their own words, tell me what the email said. Each and every conversation went something like this:

Me: “Hi, can you please read this example email?”
Food Cart Owner: “Sure, okay.”
Me: “Does it make sense to you?”
Food Cart Owner: “Yep, pretty clear.”
Me: “Great, can you tell me in your own words what it says?”
Food Cart Owner: “Um... Something about reviews, I think? I’m not actually sure.”

Not a single person understood what the email was trying to tell them. I realized that by putting the call to action button above the text that actually explained the new feature, users were probably clicking on the shiny and enticing button, missing the point entirely.

“Great, that lady with the emails is back.”

The clouds parted, the sun shone upon me, and a choir of angels sang a mighty song probably titled “Ya Blew It.” It was looking like this may have been a factor into the bounce rate being so high. We altered the copy, moved the big shiny button below the key text, user tested it again to a more successful result, made a note that next was user testing the landing page, and sent it back into the wild. After a few weeks, the data funnel became less baffling and more successful. Users were still clicking through, but this time around actually finding and using the feature we were trying to tell them about. My guardian angel that was User Testing had truly shown me the way.

A lot of my personal experience in user testing comes from reading about it, and then trying it for myself (like in the story above), while being shaped by two factors:

Time: I didn’t have much of it! My team needed a super fast turnaround for the feature.
UX Resources: I didn’t have a UX team to help or guide me with testing, so a ton of “flying by the seat of my pants” was happening.

If you are like me and find yourself in a similar situation, do not despair! My goal is that by the end of this article, you will feel like you have the tools you need to get out there and quickly test your product on your own.

What Is User Testing, and Why Should I Do It?

At a high level, user testing is putting a product in front of a human (ideally someone who would be a potential user of the product) and watching them interact with it. There are many reasons you should be user testing, but here are a few I’m partial to:

It allows you to quickly evaluate your product: When you put your product in front of a user, it’s instant feedback on so many things; design, usability, feature excitement, and so on.
It helps you get to know your users: You’d be surprised how much more you can learn about your users when they’re semi-absentmindedly clicking around on stuff. They’ll tell you about buying habits, their likes, their dislikes, how they learn about related products, etc. It’s a great way to get to know the types of users your product will interact with, which in turn helps you start to generate a better idea of the user personas you’re trying to cater to (if this is something that you have not done already).
It offers a new perspective: Someone outside your team who doesn’t look at your product every day is more likely to see things that you and your team are missing.
Real talk: It’s highly likely that nobody else at your company is doing it. But hey, if they aren’t UX, it’s probably not within their purview. However, if you’re like me and find yourself in a situation where you do not have a User Experience team to help with user testing (or your company does not have the means to hire a consultant/firm/UX designer), then you’re next in line to do that, my friend. (Even if you work with a UX team, I implore you to at least participate/observe the testing.)

Okay, but, Where Does One User Test?

One of the many neat things about user testing is you can do it almost anywhere! Here are some examples of where I’ve done user testing:

At the food carts in downtown Portland
In numerous coffee shops (personal favorite, I love coffee)
At a bar in Reykjavik
On the bus during my morning commute
On the top of Jungfrau

Basically, wherever there is another human being besides yourself, you can user test!

Google history: “how do i test a user,” “where do i find a user,” “do sharks have taste buds,” “user near me”

How Do I User Test?

Well, here’s the catch -- it usually requires talking to strangers. Terrifying, I know. The good news is, I would argue that this is the hardest part of user testing! Talking to strangers is uncomfortable, and most humans go out of their way to not be uncomfortable, but that discomfort is for a very short amount of time.

Find a user to test.

This is probably the second-hardest part. If you’re looking to test on any ol’ human, then just walk outside and talk to the first person who saunters by. However, if you really want to test someone who falls in your target market, then you need to be a bit more selective. You could start hanging around areas where your target market also hangs out, or posting an ask in a social media group curated for your target market.

For example: In the story I told you earlier, I specifically needed to know why small business owners (aka our target users) weren’t doing what I wanted them to. In this case I didn’t really want just anyone, so I went to the food carts (where the business owners were very likely present) during slow hours, and user tested there. That way, I could be confident that I was getting feedback from the types of users who would receive the problematic email.

Ask them to have a look at your product.

Some people will say no (can’t win 'em all), but most will say yes. Sometimes that requires a bit of bribery, like buying them a coffee or some kind of baked good, but what I’ve found is most people really like telling you their opinions about something. Like... a lot. It’s borderline weird.

Observe!

Set some loose expectations. For example: “We’re working on a product for [thing]. I’d like you to first look at the home page, and tell me what you think before you start exploring.” Try not to be too rigorous with these guidelines, because you don’t want them to potentially impede valuable feedback from the user.

Ask open-ended questions. In addition to this, I also try to avoid starting questions with “Why.” I imagine a solid 60% of you are internally exclaiming “BUT WHY?!” as you read this. Starting a statement with “why” tends to put folks on the defensive, because the word itself can be rather provoking. Instead of why, try using:

“What made you click there?”
“I noticed that you clicked on the same button 14 times; what are you expecting that button to do?”

Let the user wander. If they’re human, they probably will. Even if you’ve given them what you think is a direct ask like “look at this homepage and tell me what you think” and they start clicking all over the place, let them do it for a while before gently placing them back on course, but remember to observe, and ask them about what they’re doing.

TAKE NOTES. Most of us can’t remember what we wore yesterday, let alone all the little details of user testing that happened mere hours earlier. If you’re able to record the session, great, but if not, notes will be your lifesaver. You can take them back to your team to share your observations later.

BONUS POINTS: Bring a buddy with you!

Some things are just easier when you’ve got a partner in crime, and user testing falls into that category.

This User Testing Buddy can be really any of your coworkers, but I’m a huge fan of bringing a developer along. It gets them out of the building (which they never get to do), and also gets them rare, precious interaction with the user (which they also never get to do but almost always enjoy doing). I’ve found that this rare face time with users often helps the developer’s personal investment in the product, while providing fodder for ideas to improve or expand on said product. Everybody wins!

“We did it! We talked to strangers!”

What if I Don’t Have Anything for the User to Test?

No problem! You don’t need a prototype to learn more about the people using your software. Going out and finding users in your target market is still a great opportunity to learn about their pain points, and how they’re solving (or not solving) those pain points now. This will also help you start to get an idea of the types of personas your product should be targeting.

Before I started to work on requirements for building a Magic: The Gathering app, I went to card shops and Magic: The Gathering Facebook groups and asked people how they organized their massive card collections (aka, the problem). I got a ton of feedback (including users identifying pain points), and from that feedback, started to see types of users who solved the problem in similar ways (which helped me create personas).

When Should I Be User Testing?

This sort of depends on whom you ask, but in my experience the general consensus from folks who’ve been doing this much longer than I have is “test early, test often.” If you can’t do that, or if you missed the “test early” boat, it’s better to test at all (even if the product has been out for a while) than never!

Remember: Sharing Is Caring

Review your observations to the team at stand-up the next day, or type them up and share them in a document with your stakeholders. Keep it somewhere that’s easy to access in case you want to review observations in a planning session. Think of it as another way you can evangelize for the customer.

“It turns out the real treasure was the user testing we did along the way!” - Everyone who user tests

Round It All Out with Data

And here you thought I would let you get out of here without some kind of reference to data. You must not know 'bout me.

In my experience, data is the proverbial peanut butter to the jelly that is user testing. It was thanks to data that I even knew something was amiss with the emails, and that I should probably go user test to investigate further. While it’s not necessarily the worst thing in the world to simply hand your product to someone and watch them interact with it, if you know what areas your users are struggling in, you can then tailor your user testing to target those pain points.

At Kongregate, we use Amplitude to look at how our players are interacting with Kartridge, and while it’s quite possibly my favorite data companion I’ve yet to use (Pendo as a close second), data can only tell so much of a story. User testing can help round out data that you’ve collected and are looking at. For example, if you’re noticing users taking a specific path in your data software but aren’t sure why, watching a user flop around in your product could help add some clarity.

User Testing Adventurers, Beware!

While the benefits to user testing are seemingly endless, be careful to not get caught in these pitfalls:

Testing users who are not your target market

For example, if you’re building a website to help a user find the perfect pair of jeans, and you user test someone who doesn’t believe pants are real, their feedback might not be the most helpful to you (for a number of reasons).

Testing users too close to the product (support, sales, other people working on the product, etc.)

I’m not saying “don’t get feedback from internal stakeholders,” because you definitely should (this is called “dogfooding,” and done correctly, it can be a helpful practice); just be careful to not test them exclusively. Your support and sales teams probably already have an idea of what they want the product to be, and they can speak with authority because they likely interact with one or two of your customer personas on the regular. While their input is valuable, remember it’s one piece of the puzzle, not the entire picture.

Testing with an ulterior motive, or to “prove” a point

User testing, like any other kind of data, can be all too easily manipulated to say something you might want it to say, instead of what it just is. So if you find yourself in some kind of professional war with the office Dinkleberg, I recommend (for a number of reasons) finding some other way to bury the hatchet instead of manipulating data you gather from user testing to get your way.

Most importantly, please remember: You don’t know what you don’t know

Unless you have a lot of experience in user experience and design, you should always keep in mind that you are sticking a metaphorical toe into a vast metaphorical ocean. I recommend reading, listening, and talking to folks with a lot of experience in testing and research to further hone user testing for your product. If your company has the funds, hire a consultant or a firm, or even fill an entire new UX position with someone who’s got the experience. And of course, practice!

Helpful Resources to Get You Started

“Don’t Make Me Think!” and “Rocket Surgery Made Easy” by Steve Krug: My most favorite introduction to user testing in book form that I’ve ever read. Mostly because I love Mr. Krug’s sense of humor, as it makes the book super easy and fun to read. Start here!

Product Talk: Ran by Theresa Torres, a firm believer in user testing. Her site adds valuable Product Manager insight, such as Product Discovery.

UserOnboard: This website, run by Samuel Hulick, examines how popular apps and websites handle their signup experiences. It’s a cool opportunity to watch an experienced UX designer’s train of thought.