Wakefly BlogA Beginner’s Guide to A/B Testing

What is A/B Testing?

A/B testing, also known as bucket testing and split-run testing, is a randomized experiment in web analytics, which has two variants/splits: A and B. In the digital realm, A/B testing is about splitting your user base between two variations of a view or web page and observing which is more effective in your business conversion goals.

The variant can be as simple as applying a new color theme, adding.changing text, or sometimes it is a whole redesign of the page. The original version of the page is usually called the “control”, and the modified version is known as the “variant”. You then direct a portion of your visiting traffic to the original and the rest to the variant.  The winner of the test becomes the “Champion”.

Why should you A/B test?

Most A/B tests are performed to increase the conversion rate and achieve business goals. Even the most prolific marketer is just guessing at what will resonate with their users.  A/B Testing gives the marketer the opportunity to try different approaches and find that unicorn that makes conversions soar. The worst thing that you can do is launch your website and not change it again.

How to Perform an A/B test?

Most A/B tests are completed using 3rd party tools that work in conjunction with your website.  There are several options available. However, how you approach your test should generally be the same:

1. Review your metrics to identify opportunities for improvement.
2. Talk to colleagues, sales, customers, prospects to develop some ideas to test.  Prioritize.
3. Create a hypothesis about the results that your idea will bring.  Know exactly what you are trying to accomplish with your test. Be as specific as possible.
4. Decide the page and element that you will test with.
5. Determine the variant that you are going to create.
6. DOCUMENT EVERYTHING.  This will be useful when you are evaluating whether the test works or not.  It is also helpful if you need to go back and look at old tests to remember what it is that you were testing.
7. Run the test.  Be sure to accumulate enough data over an appropriate time period.
8. Compare results to you hypothesis.  Choose your Champion.
9. Repeat.

What should you be Careful About?

Although A/B testing is a relatively simple concept, there are pitfalls that could cause your test to backfire. Here are a few that you should be careful about:

• Make sure your testing period isn’t too long or too short.  For example, if you are a B2C business, make sure you run the test to include seasonal or daily differences.  You may get “enough” traffic on Monday to consider your test complete, but you need to run it a full week to accommodate the differences in daily purchasing habits.  Alternatively, you don’t want to let it run forever.
• If you are performing URL redirects to run an A/B test, use a 302 (temporary), not a 301 (permanent) redirect. This will let the search engines know that the redirect is temporary and the original URL should be kept indexed.
• To keep the test conditions equal, always test both variations (the control and the variant) at the same time. For example, if you test one version for a few days, then the other version after that, you won’t be able to confidently say which version was the true winner since the environment of both were different.
• Don’t try to test too many variants at one time.  You won’t be able to determine what change was the main cause for the result.
• Be careful making large scale changes to other parts of the site that you aren’t testing.  Those changes could impact your results as well.

Unfortunately, many people get frustrated with the time that it can take to complete a testing scenario and try to rush through the steps.  Please don’t. Slow and steady will definitely win this race.