A/B Testing

A/B tests, also known as split tests, allow you to compare two versions of something to learn which is more effective. Simply put, do your users like version A or version B?. The concept is similar to the scientific method. If you want to find out what happens when you change one thing, you have to create a situation where only that one thing changes.

Share
Format Method/ICTTimeframe 1 weekGroup Size 1-100Facilitation Level MediumRequired Materials PC, Internet connection

Steps

  • Identify a problem. Make sure you identify a specific problem. “Not enough conversions,” for instance, is too general. Many factors determine whether a website visitor will become a customer or whether a recipient of an email will be interested to click on the website. That is why, you need to know why your material is not converting.
  • Analyse user data. Technically, you could conduct A/B testing on everything that your customers see when they open your emails, but that would take a lot of time. There are a lot of design and content elements that they encounter that probably aren’t relevant, so you need to figure out which element to target.
  • Develop a hypothesis to test. Now you are really narrowing it down. Your next step is to decide exactly what you want to test and how you want to test it. Narrow your unknowns down to 1 or 2, at least to start. Then you can determine how changing that element or elements might fix the problem you are facing.
  • Conduct the test. Develop a new version of the test item that implements your idea. Then run an A/B test between that version and your current page.
  • Analyse the data. Once the test is over, look at the results and see if the new version of your item resulted in any noticeable changes. If not, try testing a new element.
  • Find new challengers for your champion. The A/B testing world sometimes uses “champion” and “challenger” to refer to the current best option and new possibilities. When 2 or more options compete and one is significantly more successful, it\\'s called the champion. You can then test that winner against other options, which are called challengers. That test might give you a new champion, or it might reveal that the original champion really was the best.
  • Once you have run through all 6 steps, you can decide whether the improvement was significant enough that you can end the test and make the necessary changes. Or you can choose to run another A/B test to evaluate the impact of another element, such as the size of the button or its colour scheme.

Benefits

  • Evaluating the impact of changes that are relatively inexpensive to implement.
  • A/B testing is not only cost effective, it's time efficient. You test 2 or 3 elements and get your answer. From there, it’s easy to decide whether to implement a change or not.

Tips

  • Use representative samples of your users. Make the groups as equal in size as you can and—if you have access to the data—evenly distribute recipients according to gender, age, and geography. That way, variations in these factors will have minimal impact on your results.
  • Maximise your sample size. The more people you test, the more reliable your results will be. This ties into a concept that statisticians refer to as “statistical significance.”
  • Let the test end before making changes. Because A/B tests let you see the effects of a change in real time, it's tempting to end the test as soon as you see results so you can implement a new version right away. However, doing so means your results are likely to be incomplete and are less likely to be statistically significant. Time-sensitive factors can impact your results, so you need to wait for the end of the testing period to benefit from randomisation.
  • Run tests more than once. The only way to make sure your results are accurate is to run the same test again with the same parameters.
Download Tool Supporting Files Not available