A/B Testing Vs. User Testing: Helping You Save Time

You have a new product, website, or content that’s ready to launch, but there’s disagreement about how it should look.

To settle the dispute you start thinking about a/b testing, but you are under a crunch to get this out very quickly. User testing offers a faster turnaround, but which test is better?

The better question is: how much time do you really have?
A:B Testing vs. User Testing


Two Types of Testing Defined

A/B testing is a term used in marketing and business intelligence for a type of controlled experiment where two variants of a web page are observed to see if they affect a user’s behavior. The variant that achieves a higher outcome of interest, click-through, sign ups or purchases, is the variant that you would channel all of your traffic through.

User Testing, or Usability testing, is a technique that observes a single user’s interaction with a website or product to see if it achieves its intended purpose.

Before you jump to either method, run through these best practices and be honest with yourself about how much time you have.

A/B Testing: All the Time if You Have the Time

Smaller changes for smaller gains over a longer period of time are the name of the game in a/b testing. If you are impatient or your stakeholders are impatient, a/b testing is an uphill battle. Set an expectation that these take time, research and will result in small percentage changes.

Here are 5 key things to remember about a/b testing.

1. Don’t a/b test unless you have sufficient time (weeks)

Impatience is an a/b test killer. A recurring theme I see is that a test will be called before it achieves statistical significance. If your stakeholders are demanding results now, skip ahead to the user testing.

2. Develop a solid hypothesis before a/b testing

Make a statement based on the limited evidence available to define your variant. Try not to guess a solution at a glance. Review your traffic flow and check your page bounces and where that traffic exits.

This can be time consuming, but it’s more wasteful to a/b test just for the sake of a/b testing.

3. Be wary of false positives or immediate success

It can be very exciting to see a variant winning in the first few days of your a/b test.

Who doesn’t like the feeling of being right?

But it’s good to be overly critical of any success that comes before the a/b test has reached a statistical significance.

4. Expect small gains from a/b testing

Need a 10 to 15% bump in conversion? Its unlikely you will achieve that in one a/b test. Consider user testing instead. It’s reasonable to see 2 to 4% increases with a successful a/b test.

Just remember these are long plays and if your site is already converting well then these small gains are great victories.

5. Don’t rush an a/b test!

I keep repeating this because it’s such a common problem. You see a variant crushing it and want to end the test, or one variant looks like a resounding failure in the first few days.

Resist temptation! There are many external factors that may be affecting the results, and your traffic may fluctuate wildly throughout the week.

Check the calendar, make sure there wasn’t a large holiday you forgot about. Your website has a traffic rhythm that crests and falls regularly. Find the edges of this cycle and set the timing of your a/b test to it. Then stick to it through the end!

User Testing: Great For Short Turn Around Changes

Maybe the issue is that a common sense change that’s difficult to see won’t ever surface using a/b tests. Sometimes the greatest gains will be from watching someone actually interact with your forms and workflow.

You’d be amazed by the easy wins you can find when you watch a user click through your sign up workflow. For months you’ve been tweaking words and colors and then a user test suddenly points out that your form is confusing and wishes it guided him easier.

Conducting a user test first can often save you time and eliminate the need for future a/b testing.

Follow these 5 best practices to get the most bang for your buck:

1. Define the goal of your user test

This seems simple but it’s very easy to get lost in user testing with all the little things you see that don’t aid your defined goal. If you are testing to see if users can easily purchase your product, that’s the only outcome you should be worried about.

2. Make sure your user test tasks are clear and easy to understand

User tests can be a powerful learning tool, but they can be very expensive and time consuming if you have haven’t laid out your tasks carefully. Run through your test yourself and be explicit with your instructions.

Nothing is worse than reviewing a user test where the subject is lost because of vague instructions.

3. Stay focused on your goal when viewing tests

Testers are instructed to narrate their entire experience while testing. They will talk about everything they see, like and dislike. It’s easy to get lost in the details they mention that won’t necessarily help your end goal of making your user experience simpler and easier to understand.

Take notes, but don’t act on everything they say. Comments that are unrelated to your current goal may turn into future user tests or a/b tests.

4. The solution will probably be more obvious than you expect

You will be surprised to find the things that need to be changed will be so obvious to the users, but not to your team. For example, the tester reaches that critical moment and declares, “Where’s the button to pay?” or “Why can’t I click this?”

We look at our product or service so often and run through our work flows so quickly that we become blind to things that are obvious to a new user, and vice versa.

5. Implement changes immediately

Don’t hesitate to implement these small changes bravely and quickly. With a enough traffic, you will see rather quickly if your changes to your forms and workflows had an impact. But unlike a/b testing, remember to monitor the changes you implement after user testing closely in case you must revert back.

Know your stakeholder’s urgency and choose the right test

My organization leverages user testing all the time. And the reason is that my stakeholder, the CEO, wants to iterate rapidly and adjust to the results quickly. Fail fast and adjust is his motto.
Take the time to understand your stakeholder’s desired speed of iteration before applying a/b testing or user testing to website changes or product launches so you can be sure you’re using the right tool for the job.

Other Articles You Might Be Interested In:

get the agile marketing styles ebook

About the Author:

Joshua Robitaille

Leave a Comment