A/B Testing Explained + a Collection of A/B Testing Learning Resources

In this article, we’ll take a look at A/B testing, how and when it’s conducted, why it’s important, what the advantages of A/B testing are, as well as look at some of the excellent reading material and learning resources that would help extend your knowledge and get a better understanding of the A/B testing concepts.

What is A/B testing?

A/B testing (aka bucket-tests or split-run testing) is a randomized experiment with two variants, A and B, which includes the application of statistical hypothesis testing or two-sample hypothesis testing. The goal of the A/B testing is to compare two versions of a single variable by testing a subject’s response to variant A against B and determining which one is the most effective.

To explain a little further, you compare two versions, A and B, of a single variable, which are identical except for one variation that might significantly affect the outcome, say, user behavior. For example, A might be the currently used solution, in this case, it’s called “control,” and B is a modified version of an existing solution, in this case, it’s called “treatment.”

Why is A/B testing important?

Testing different variables and changes that you make to your project is crucial, otherwise, you’ll never know how effective your changes are or what has essentially caused the difference in the outcome. But A/B testing can do a lot more than prove how changes have impacted the outcome of your solution or, say, marketing campaign or conversion rates. Speaking of A/B testing used in user experience research and website marketing campaigns, it has long-term benefits that ultimately lead to an improvement in user engagement, better content, reduced bounce rates, higher conversion values, and increased sales.

A very detailed explanation of A/B testing from Human Computer Interaction Seminar at Stanford:

Advantages of A/B testing

A/B testing is cost-effective and time-efficient. You run 2-3 tests at the same time and get an answer to your question or business problem in the shortest time possible, from there it’s most probably the matter of a single decision whether to implement a change or not, and if something doesn’t seem to work, then it’s easier to revert back the changes and come back to your earlier solution that seemed to work better. Trying to figure out what increases your conversion rates as early on as possible, will eventually result in less money and time spent on marketing campaigns.

What to test with A/B testing?

There’s much to evaluate with A/B testing, especially when it comes to user experience and users facing content on your site. For example, A/B testing works perfectly well on email campaigns, multimedia marketing strategies, paid advertising, website design. In each of those categories, you can A/B test a number of variables, such as color schemes, layouts, images, headings, pricing, special offers, and, of course, call-to-action button designs. Moreover, almost any style or element on the website is testable with A/B testing.

When to do A/B testing?

There are some instances when you absolutely have to perform an A/B testing. These must-perform cases include redesigning of a website because redesigns can result in the unfortunate loss of sales or lower conversion rates; when you change a service, a feature, or a plugin, especially when the upcoming changes affect the customer data or purchase process; when you decide to change the prices; when you think your conversion rates are insufficient; when you want to increase revenue. Overall, whatever changes you want to make to your service or product, the best thing is to always test.

Different types of A/B testing

First of all, there are concepts such as multivariate testing and multinomial testing, which are similar to A/B testing except that instead of two variants, you’ll test more than that or use more controls. The reason behind testing more variants is because simple A/B testing is not suitable for observational, quasi- or non-experimental situations (which is most common with survey data, offline data, and other more complex solutions).

Another existing differentiation, which is attributive to marketing campaigns, is on-site and off-site testing. For example, for an on-site test, you’ll want to think of all the things that are related to your website like sales-related pieces like prices; and for an off-site test, it could be an ad or sales email that you’ll be testing.

A/B testing tools

There are multiple tools that you can employ during your A/B testing. Preferably though, that you’d conjure tools of your own, like a proprietary system that would be totally and fully customized to your needs. If you’re not in a position of doing something of your own, these are a few companies that can help you with mastering A/B testing or doing A/B testing for you:

• Google Analytics & Google Optimize, which is a free option for almost all A/B testing that you can perform on your site. However, if you run a larger business, you’ll probably need to sign up for Google Analytics 360 suite. The only downside with Google packages might be the need for a professional who really understands the tools and knows how to work with them, preferably engineers who will set up advanced trafficking and customize other features that would suit your business.
• VWO, which is trusted by companies like Target, and Virgin, is an A/B testing optimization platform that is specifically tailored for bigger companies. Their suit includes A/B tests, split URL tests, and other multivariate tests. VWO has a pretty straightforward pricing model but overall it’s a very similar platform to Optimizely, which we’ll cover in a moment.
• Optimizely is an experimentation platform for marketing, engineering, and product design teams. The suite includes a powerful experimentation A/B testing tool that allows you to run multiple experiments on one page at the same time and test various variables of your design. Optimizely is one of the leading CRO platforms which offers far more than A/B testing, but if that’s A/B testing you’re really in need of, then consider using Optimizely Web which provides features A/B testing and other features for optimizing web pages.
• Omniconvert is a conversion rate optimization platform that offers A/B testing functionality as well as many other features like personalization, overlay, survey, segmentation tools with over 40 segmentation parameters, and so on.
• Crazy Egg is an optimization software that also offers A/B testing, heat mapping, usability testing tools; moreover, you can perform A/B testing without any coding experience.
• AB Tasty is another conversion rate optimization software that offers A/B and multivariate testing, personalization and marketing tools, and data insights among the others. The AB Tasty visual drag-and-drop editor allows you to build and run A/B tests, split tests, multivariate and funnel tests.

A/B testing examples

Let’s make a few examples of A/B testing from some simplified examples to more complex ones.

For example, in terms of email marketing, you’ll send out an email of version A to half of your subscribers and an email of version B to the other half. Then you’ll measure the performance of each and check it against your conversion rate goals. The examples of data that you can gather and analyze are click-through rate, completion of a form, a purchase made, etc. Splitting or, otherwise, bucketing offers is suitable for every time you want to find out what works best and what ultimately lands you with a sale.

Another example could be a couple of versions of a blog post that you want to find out works best and read more or commented on more. For example, if you want to evaluate the font and the percentage of images that make the blog post attractive to the readers, then you can create four pages, each displaying blog post with: Arial font with images, Arial font without images, Open Sans with images, Open Sans without images. A/B testing software usually takes care of all of this for you rendering the parsed results for your marketing team to decide what works best for the company’s goals.

Let another example be connected with the so-called social proofs, which are centered around a company’s proof mechanisms like testimonials, star ratings, industry authority badges, client logos, etc. So what you can test in terms of social proofs are the performance of some of your products with customer reviews and those with no reviews at all, testimonials with your brand logos and those without, testimonial A vs testimonial B, client logos vs industry regulator logos, and various design-based experiments, and so on.

How to do a/b testing // how to perform a/b testing?

The process is relatively straightforward and reminiscent of an engineering process, where you have to identify a problem first, then analyze your data, develop a hypothesis, and finally, test it, and reiterate if necessary. Let’s break down the process a little further:

Problem identification

In this stage, you’ll need to identify a very specific problem (something too general like “we don’t have enough conversion rates” won’t work). For example, you find out that your email campaigns are not converting, users open up their emails, but a few actually follow the links and buy the stuff off your site.

Data analysis

Herein, you’ll have to analyze the specific elements that might affect your campaign or be relevant. Speaking of relevance, there would be a lot of design and content elements that users interact with or pay attention to, yet very few of them would play a crucial role in conversion. Your task is to find out those elements. Let’s elaborate on the previous example a little further: your users open up your emails (nothing wrong then with how you write your headlines), they spend time actually reading through them (nothing wrong with how you write your copy), there’s also evidence that users buy your products through other channels (nothing wrong with how you present your products), so there must be something wrong with either a call to action button or presentation of information that calls your users to act.

Development of a hypothesis

Now, when you’ve analyzed your data, you can narrow down your unknowns to one or two specific variables that you’d like to test. In the example above, most probably the call-to-action button needs to be changed: for example, instead of a yellow button below the fold, you want to bring it to the top of the screen and make it a different color.

Conduction of an A/B test

Now, develop a new version of your idea and test it against your old version. As in the example above, create a version of an email with, say, a blue button on top of the screen, and test it against an older unchanged version where the CTA button is yellow and hidden below the fold.

Analysis of received data

Analyze the conversion rates or any other results you were expecting to find out through the test, and see if you can notice any substantial changes, and if not, then you’ll most probably have to repeat the whole cycle or try and test another element. For example, your new button placed above improved the conversion rates but only ever so slightly, so you decide to add another button in the middle of an email, so you’ll have two buttons now, that is above and in the middle. Repeat the test and analyze the results against the new winner. Speaking of terminology here, a winner or a champion is the current best option among the possibilities, a challenger is another option that you’d think has a potential of outrunning and beating a champion.

A very insightful talk on A/B testing from the team at Etsy

How to learn A/B testing: list of A/B testing learning resources

We’ve gathered a collection of resources for you to learn more about A/B testing and how to properly conduct it.

General information

Before you embark on a journal of statistical research and analysis, you’d better familiarise yourself with the basics by reading your industry-specific literature, articles in business magazines, or those written by the industry experts as opposed to amateurs.

• HBR has a great article on A/B testing including some of the most prominent examples that demonstrate the efficacy of testing as well as explain the importance of conducting A/B tests in establishing causality.
• Other great sources of mostly theoretical information would include books written on the subject, such as A / B Testing: The Most Powerful Way to Turn Clicks Into Customers and You Should Test That: Conversion Optimization for More Leads, Sales and Profit or The Art and Science of Optimized Marketing

Conducting the test and the basics of statistics

Knowing the “whys” and the importance of conducting the tests won’t get you anywhere as long as you don’t answer the questions like “what metrics to use,” “how to estimate the sample size,” “how to split the test audience,” “what statistical test to use to calculate the significance level,” “how to run and how to interpret the results.” To find out how the actual tests are performed and what specific statistical knowledge you’d be expected to have, head on to a few resources we’ve outlined below.

• Udacity offers an excellent course on A/B testing developed by Google which covers the whole ground-up process of hypothesizing, designing, and analyzing tests.
• “Controlled experiments on the web: survey and practical guide”, a paper written by Ron Kohavi from Microsoft, is an absolute must-read, which many reviewers have found to be even better than any other written material or book on the topic: it gives an overview of the problem, covers in great detail the controlled experiments performed, a lot of mathematical data with explanations and reasons for using each, and finishes with a comprehensive analysis of the test outcomes.

Industry-specific tests

Now, if you want to learn more about the tests that are performed in your specific industry, the terminology that’s applicable to your business, and find out the intricacies of doing A/b testing in your field, then these are the resources that you might find of value:

• A/B testing in Social Networks
• A/B testing integrated with Engineering Rollout Phases
• A/B testing in media and publishing 1 and 2
• A/B testing in e-commerce
• A/B testing in marketing
• A/B testing in B2B

Advanced statistics

To learn the difference between various approaches to A/B testing, including the differences between the traditional t-test or fixed horizon testing and sequential testing based on Bayesian statistics, you’ll need to get a good grasp of both concepts, especially the sequential testing, as it’s a relatively new concept with still a few valuable resources covering the subject in detail.

• Bayesian Machine Learning in Python: A/B Testing Udemy course
• The story behind Optimizely’s New Stats Engine

Experimentation Platforms

Huge corporations run experimentation platforms that enable building, planning, launching, and scaling A/B testing, those platforms, however, require significant investments both manpower- and money-wise. If you’d like to learn how big corporations are running their experimentation platforms then these are a few good starting-point reads to familiarise yourself with the concepts.

• Airbnb’s Experimentation platform
• Uber’s Experimentation platform
• Facebook Experimentation platform

A/B testing in machine learning

These are a few resources if you want to learn more about applying machine learning concepts in A/B testing

• A/B Testing with Machine Learning – A Step-by-Step Tutorial
• Accelerating A/B Testing with Machine Learning
• A/B testing vs Reinforcement learning

FAQ:

What is a/b split testing?
A/B split testing is the same as A/B testing, the definition of which we’ve given at the beginning of this article.

Where to use A/B testing?
A/B testing can be virtually used anywhere where you have to determine the best route for conducting business: marketing, advertising, data science, user experience, and so on.

What does “variant a” in a/b testing usually represent?
The variant A usually means the currently used solution, called a “control,” whereas the B variant is a modified solution of an existing “control,” called a proposed “treatment.”