How do I perform a successful A/B test?
A/B testing is becoming increasingly important. This is because your customers expect everything to become more and more personal. So what's an easy way to make your newsletter more in tune with your target audience? Exactly: A/B testing! Test what exactly your target audience wants to see in the newsletter. In this blog, I explain how to conduct a successful A/B test.
What is an A/B test?
Thanks to A/B testing, it is possible to split your newsletter into two variants: Variant A and variant B. Here you test whether it makes a substantial (conversion) difference to adjust something in your newsletter for the target group. An A/B test is created when you have a hypothesis. An A/B test can be used to find the answer to important questions in your company, increase the turnover of your existing visitors and lay the foundation for a data-driven marketing strategy. A/B testing actually belongs in your marketing strategy.
How an A/B test works
In our A/B test, you show part of your newsletter subscribers variant A and another part variant B. The version that generates the most clicks, opens or unsubscribes is the winner. The group of newsletter subscribers who remain get the winning version in their inbox. This version then becomes the new control version and you get to work designing a new one. You keep on testing until you are satisfied with your end result, such as your conversion rate, click rate and so on.
However, we would like to warn you not to focus on conversion alone. If you are going to do an A-B test on conversion only, it is logical that "the cheaper your product, the higher your conversion" would be. Which leads to free products. Not recommended. 🤑
What do you want to use A/B testing for?
That is a good question! It differs per web shop what is useful for A/B testing. We would love to give you a list, but there is no 'magic must A/B test list'. The tests are done with your own data, such as customer data, segment data etc. Take a good look at your collected data and see from there where it is valuable to A/B test.
How do you ensure a successful A/B test?
In order to carry out a successful A/B test, preliminary research is required. First you set goals, analyse your goals, come up with a hypothesis, then you design and only then you start A/B testing. That sounds like a lot of work, but believe us: it is not that bad. And in the end, you get a valuable and well thought-out A/B test in return.
1. Setting goals
Set goals for yourself, exactly what you want to achieve with your A/B test. Would you like more newsletter subscribers? Or maybe more conversion? With a clear goal in mind, you are sure to have a good start for your A/B test.
2. Analyse
After you have set your goal, it is time to analyse your current results. We also call this your 0-point. Because: how do you know if your test is successful, without knowing how you stand now? Exactly: you do not know. That is why it is important to first look at what your results are now. On the basis of this analysis, you formulate a hypothesis.
3. Formulate a hypothesis
Before you start an A/B test, you must first have a hypothesis. A hypothesis is measurable and aims to solve a specific (conversion) problem. In an A/B test, you test a hypothesis and not an idea.
You can use Craig Sullivan's Hypotheses Kit for this:
- Because I have found that [data and/or feedback from previous research]
- I expect that [the change/improvement you want to test] will have [the impact you expect] and
- I will measure that impact using [the data measuring tool you will be using].
Fill in the blanks and your test idea is changed to a hypothesis. This ultimately makes it easier to read your results from your A/B test. Now that you know what the goal of your A/B test is and you have converted your goal into a hypothesis, it is now time to draw up your design.
4. Creating the design
By drawing up a design we mean that you are now going to think about the design of your newsletter. Because whatever you test, what will it look like? If you want to change the content in your newsletter, you might need a UX/UI designer. Do you want to test the subject line, preheader or sender of your newsletter? This also comes under the heading of design. Define somewhere what you are going to test and how you are going to test it.
5. The magic of A/B testing
Hurrah! Finally, but still: this is where A/B testing really begins. In this step you create your A/B test the way you envision it. That is not difficult anymore: you have a hypothesis and a design in mind. Now it's just a matter of putting the A/B test together. In Reloadify this is very easy, since variant A is copied into variant B. This way you never have to create the same newsletter twice. Handy!
6. The evaluation of the A/B test
You've done your homework nicely: you've set goals, hypothesised and performed an A/B test. That's it. Right? Nothing could be further from the truth. Because after you have carried out your A/B test, perhaps the most important thing comes: becoming wiser from your A/B test results. Because why did the winning test win? And why did the losing variant lose? Chances are, for some segments your losing variant is actually the winner - information you can use to your advantage now and in the future. The analysis is about much more than just looking at which variant won or lost. Also take a look at the individual segment data, for example.
How to save the results of your A/B test
Suppose you take your first test tomorrow. In a few years' time, will you remember exactly what the result was and why? Probably not. That is why you should save your A/B test results. Without a good test archive, all insights are lost. In addition, there is a big chance that you will perform the same tests if you do not save everything properly.
In principle, you can set up your archive however you like, as long as it contains the following elements:
- Your hypothesis
- Screenshots of your A/B test results
- Which customer won/lost
- Insights from the analysis
Over the years, you will thank yourself for keeping this archive.
Learn More
Did you know that at Reloadify you can also get a dedicated IP address? You need your IP address to send out your emails. Not just you, several clients use the same IP address at Reloadify. This is also easy. If you have more than 50,000 profiles and mail them several times a week, we recommend you generate your own IP address. Are you interested in your own IP address? Then please contact us.
