Email marketing is affordable, easy to set up (even for beginners), and effective for both business-to-business (B2B) and business-to-consumer (B2C) businesses. But, I’ve witnessed many instances where email marketing ultimately fell short of expectations and metrics. In most cases, it was because of simple details, like a wrong subject line, offer, or call to action—which is something you can identify through email A/B testing.
A/B testing is the practice of sending two different versions of an email to segments of your contact list to see which is most effective. Email marketing can be very variable; the littlest details can determine its success or failure—which is both an advantage and a challenge for small businesses.
Here are some of the small details you can create A/B tests for that actually make significant impact on the overall effectiveness of your email marketing campaigns:
- Subject lines: longer vs shorter subject lines, using or not using emojis, and so on
- Using a “real person’s” name or a generic email address in the sender field
- Personalizing the greeting line of your email with the recipient’s name
- Including vs not including images and media in your email content
- The overall tone of your subject line and email content (friendly vs formal, etc.)
- Sending at one time of day vs another (for example, morning vs evening, weekday vs weekend)
These are just a small sample of all the little variations that determine the success of your email campaigns. There isn’t necessarily a universal right or wrong answer for any of them, as your audience is unique. However, that’s also why email A/B testing exists and is one of the most useful strategies in email marketing. In fact, I’d argue it’s a necessary part of ensuring your email campaigns’ effectiveness.
And, it isn’t even too complicated to set up. Keep reading to learn all about email A/B testing, how it works, plus some best email testing ideas and tips.
Overview of Email A/B Testing
Email A/B testing (also called email split testing) is testing two variants of the same email to see which is more effective with your audience list before sending out the full campaign. The test is sent to two samples of the target audience, and the more effective version (aka the “winning” version) is sent out to the entire mailing list.
The differences between each version of the email are usually subtle: a difference in subject lines, content layout, or even different sending times. These are referred to as the variables of each email A/B test. Ultimately, email marketing A/B testing gives you valuable information about your audiences that enables you to build more effective email campaigns.
7 Steps to Do Email A/B Testing
Now that you know what email A/B testing is and how it works, keep reading to learn how to set it up. There are seven steps involved, from identifying the variable to test to using the data you gather for future improvement.
1. Identify the Variable to Test
The first step for forming your email marketing testing strategy is to determine the specific variable of the email campaign you want to test. As mentioned above, specific variables of your email campaign can be more effective than others, such as a shorter subject line, a different tone for your copies, and so on. And your A/B split test will help determine exactly those email elements that work best for your target audience.
You’ll first need to identify your email campaign’s objectives. Knowing your objectives will help you define the specific variable you want to test the most. For instance, if your email’s objective is to increase sales on your online store, you might want to test which copies lead audiences to click on your link the most, and so you’ll create two different email copies and see which is most effective.
Your email testing ideas will also differ depending on the type of email blast you’re sending—for instance, if you’re only sending a welcome email with the objective of increasing brand awareness, you might test which subject line leads more people to open your email. Define your email campaign objective, then identify the variable you want to test.
Here’s an example: Say you’re launching a marketing newsletter for your cafe as part of an awareness campaign to build your brand presence. But, you’re not sure what to write in your subject line—should you use a conversational tone or a more straightforward, businesslike one? So, you decide to do an A/B test with the subject line as your variable, which might look something like this:
- Subject line A: September is here! This month, an all-new menu, just for you.
- Subject line B: View Our New September Menu
Here’s an A/B test example for two different subject lines. (Source: SendX)
Pro tip: New to email marketing? Read our guide on how to do permission-based email marketing first—it’ll give you valuable background information about email marketing before learning about email A/B testing.
2. Outline Your Metrics of Measurement
Once you’ve identified the variable to test, you’ll then need to determine how to measure the test results later on. Do this by setting measurable metrics on your email campaign to determine its effectiveness, such as open rates, click rates, or conversion rates.
Here are some important email marketing metrics to check:
- Open rates: How many people open your email
- Click-through rates (CTR): How many people click on a specific link in your email
- Bounce rates: The number of people who didn’t receive your email, either because of incorrect email addresses, full inboxes, or other blockers
- Conversion rates: How many people followed through with a desired action, e.g., a product purchase or booked appointment
- Unsubscribe rates: How many people unsubscribed after receiving your email
- Subscriber growth rates: How many new subscribers you get over a period of time
Outlining measurable metrics at the beginning of your campaign will help you determine its effectiveness more accurately later on. It’ll also give you valuable data that you can use to create better campaigns in the future.
Building on the A/B split test example above: Since your campaign objective is brand awareness and your variable is your subject line, a good metric of measurement would be your email’s open rates. Your open rates will help determine which subject line piqued your audience’s attention the most and caused them to open your email.
3. Select a Sample Size & Testing Period
After you’ve decided on your campaign metrics, the next step is to choose your A/B test’s sample size and testing length. Below are more details and email testing best practices for each.
How to Select a Sample Size
Your test’s sample size is a representation of your entire audience, and is also where the statistics part of your test comes in. The key is to get the lowest possible sample size that gives you statistically significant results—meaning, you want to get as accurate a representation as possible of your entire audience while using the smallest possible sample size.
To make things easier, you can use an online sample size calculator like AB Tasty or Optimizely, where you’ll only need to input data points to calculate the best sample size for your A/B test statistically. However, these calculators work best if you have a large audience list of thousands—if you have a smaller audience, you can transpose your results into a smaller context instead.
For example: If Optimizely calculates your ideal sample size to be 5,000 for the most statistically accurate results, you can use 50 as your sample size instead—meaning 25 will receive version A and the other 25 will receive version B.
Here are some important data points you’ll need if you use a sample size calculator:
- Conversion rate: The current conversion rate of your emails. This usually ranges between 1% and 5%.
- Minimum detectable effect: The minimum change you want to detect in your conversion rate percentage. This usually ranges between 10% and 20%.
- Statistical significance: The percentage of results that lets you determine the success or failure of your test. This usually ranges between 85% and 95%.
How to Choose a Testing Period
Not every contact on your mailing list will open your email immediately upon receiving it, which is why statistically, A/B tests that run longer will give you more accurate results. That said, your business won’t have all the time in the world to do A/B tests, so a few hours is usually enough for each. As per Mailchimp, here are the recommended testing lengths for each campaign objective:
- To test open rates: 2 hours
- To test click rates: 1 hour
- To test revenue rates: 12 hours
4. Choose a Winning Combination
With your test variable, sample size, and testing period, you’re nearly at the end of planning your A/B test. The last part is to choose the winning combination or parameters that will determine which version “wins” and that will be sent out to your entire audience list. This is essentially the “test” part of your A/B test that lets you determine a winner.
Here’s an example of a winning combination you might choose: the subject line with the higher open rate after two hours.
You’ll notice that it’s a combination of the elements from the previous steps. However, this will also vary for each email platform, as not all will allow you to set a custom winning combination—some platforms automatically determine the best-performing combination according to click or open rates.
5. Choose an Email Platform & Launch Your A/B Test
Now that you have the basic elements of your email A/B test, you’re ready to set them all up. You’ll first need to choose an email marketing platform that best fits your needs (we’ve listed a few recommendations below).
Once you have an email platform, create an A/B email test and then input your test details in the appropriate fields, including the variable you’re testing, your sample size and testing period, and your winning combination, if applicable. Once all your test details are plugged in, you’re ready to send your test.
6. Measure A/B Test Results
Of course, as with every test, you’ll also need to measure your results once it’s done. This is where the metrics you defined from step two come into play. Once your test is concluded, measure the results you get against those metrics. You can usually find your test results under the dashboard or analytics section of your email platform.
For instance, from the example above, you find from your test that Subject line A garnered a 30% open rate, while Subject line B only garnered 10%. From this, you can then conclude that using a friendlier, more conversational tone in your newsletter’s subject lines can be up to 20% more effective for your brand awareness email campaigns.
Plus, you can also take note of other metrics from your test, like unsubscribe rates or conversion rates, that can help you determine other best practices. For example, say Subject line A garnered more open rates, but you find from your test that Subject line B had a lower unsubscribe rate, and so on. All these data points are helpful for your future campaigns.
Pro tip: When measuring your results, you also need to consider their statistical significance. For example, if one email version only won over the other by a 3% margin, it means your results aren’t very conclusive or significant, so you might want to redo the test under different circumstances. Another factor is if you conduct your test during a holiday period and then apply your results to regular-day emails, it won’t be as accurate.
7. Use Data & Insights to Tweak Future Campaigns
The most important part of any test is to use your results to gain sound insights about your audience, and then apply them to future strategies. After conducting your test and recording your results, use the knowledge you gain to build more effective email campaigns in the future.
For example, following your test results from the example above, you can then continue using conversational and friendly subject lines in all your email newsletters. And, you can even build on that test further to get more precise insights: test specific content types under those subject lines to see which yield better click and conversion rates, test them at different sending times, and so on.
Like any experiment, you’ll need to go through trial-and-error to find just the right conditions for your email campaigns. Of course, make sure to record all your results. Over time, you’ll be able to build a solid email marketing strategy for all your email campaign objectives.
Email A/B Testing Best Practices
A/B email marketing testing yields many long-term benefits for your email campaigns—and there are some ways to best maximize its effectiveness. For instance, as with any other experiment, testing with various conditions gets you more accurate results. Here are some best practices to take note of when split-testing email marketing campaigns.
- Record all your results: The most important part of the testing process is recording all your results, especially for small businesses. Because most research on email marketing centers on larger businesses, the insights they yield may not always apply. This is why running your own tests and finding your own results is essential. Recording all your data gives you a wealth of information that can guide you later on.
- Test during different seasons: Time is an important variable in marketing, as consumers have different spending habits during different times of the year. What works best for your emails during March may no longer be as effective during the holiday season, when you’ll likely have much tougher competition. Stay abreast of your audience’s preferences by doing A/B tests at different seasons of the year.
- Test only one variable at a time: Tests are only significant when they yield conclusive results, which you can’t do if you test multiple variables simultaneously (e.g., testing the effectiveness of your subject line and email design at the same time). To get precise and relevant results you can count on, make sure to only test one variable at a time.
- Do follow-up tests: Once you’ve tested the effectiveness of one variable, you can refine your findings even further with follow-up tests. For example, if you’ve concluded that conversational subject lines lead to higher open rates, you can also test which audience sets they’re most effective for. Doing these follow-up tests continually will help you form your best email marketing strategy.
- Study other email campaigns: Research is always useful, even after conducting several tests. You can always get more ideas for testing and improving your email campaigns from studying campaigns from other brands (especially your competitors), and seeing what elements and strategies might work for your business. Of course, use A/B testing to test your theories.
Best Apps for Email A/B Testing
Choosing the right email marketing platform is a significant part of your email testing process, as each platform has different A/B testing features. For instance, Mailchimp lets you set your own winning combination, while Constant Contact automatically chooses the best-performing subject lines. While the best platform will depend on your own level of comfort and suitability for your needs, here are our top email platform recommendations for A/B email testing.
 |  |  |
|---|---|---|
A/B testing for subject lines, design, content, and send times in all paid plans | A/B testing subject lines and “from name” testing (all plans) | A/B subject line testing and AI recommendations in all but base tier |
From $13/month up to 500 contacts | From $7.49/month up to 1,000 contacts | From $35/month up to 500 contacts |
For even more options, visit our guide to the best email marketing software for small businesses.
Did you know? A/B testing isn’t just for email—you can use it to improve your bulk SMS marketing campaigns too. Check out our list of the best SMS marketing platforms to get started.
Benefits of Email A/B Testing (Statistics)
Email A/B testing isn’t mandatory for email marketing, but it’s so beneficial that it might as well be. The biggest advantage is that it helps you create more accurate and impactful email campaigns, and it gives you valuable insight into your target audience’s likes, dislikes, behavior, and other preferences.
The statistics below showcase why A/B testing email marketing is in your small business’ best interest:
- 33% of email recipients open emails based on the subject line alone. That means that your email subject lines have a hit-or-miss chance for over 30% of your audience. So, rather than tossing a coin, take a well-informed chance instead by doing an A/B test. Some subject line testing ideas are your subject’s tone, length, personalization, and emoji use.
- 45% of recipients open an email based on who it’s from. One of the best email testing ideas is to test different sender names. This is because 45% of all email recipients open an email depending on who its sender is, so whether you include a person’s name or just your company name in your sender field makes a significant difference in your email’s effectiveness.
- Sending at the right time can increase open rates by over 13%. According to statistics from GetResponse, email open rates can vary by over 13% depending on the time of day you send them. Morning hours have a general open rate of 33%, while later hours are as low as 19%. These can vary depending on your business’ industry, however.
- Subject line length can increase CTR by nearly 3%. The length of your email subject line also affects how many clicks it generates. GetResponses’ study also found that the number of characters in an email subject line affects its click-through rate by up to 2.95%. This is especially significant for emails with product promotion campaigns or other external landing pages.
- Businesses that do regular email A/B testing see an 82% higher ROI. If you’re still in doubt that A/B tests work, recent studies found that brands that do regular email A/B tests see return on investment (ROI) increase of up to 82%, so there’s definitely a concrete benefit to all your experimenting.
Other Email Testing Ideas & Types to Consider
A/B testing your email content isn’t your only available option, especially as content is just one part of an email marketing campaign. There are other components to consider that count toward your emails’ impact, like your recipient list, sending schedule, and overall quality and accuracy.
Here are other email testing ideas you can do:
- Audience list tests: Customer segmentation is one of the biggest benefits of email marketing—you can segment audiences based on preferences, demographics, and behaviors to maximize your campaigns’ effectiveness. But first, you’ll need to test what works best for each audience set. Do this by taking a sample audience from two audience lists, then sending the same email manually to see which is more effective.
- Sending schedule tests: Certain hours are more effective for sending email campaigns, which can vary for each business. Promotional campaigns may be more effective during weekend evenings, while brand awareness might be better for weekdays, and so on. For this test, set up different campaign types as your A and B variables, then send them at the same time to determine which is more effective.
- Customer personalization tests: Customer personalization is when you include your recipients’ names in your email campaigns, which can be more or less effective depending on each business. Beyond testing whether or not this strategy works for your business via an A/B test, you can also test which type of campaign it’s most effective for, like promotional campaigns, re-engagement campaigns, and so on.
- Email text preview tests: Email preview text appears right after the subject line in your recipients’ inbox, and can play almost just as relevant a role in getting your emails opened and read. Ideally, do A/B tests for your preview text as a follow-up test after determining the most effective subject lines for your emails. They can help refine your content even further.
- Landing page tests: Email campaigns are great for promoting external landing pages like your online store or a special promo code—but only if your landing pages actually get people to convert. If you have a landing page to promote, create two different versions and then test them on the same email audience sample to see which leads to more conversions.
- Quality and accuracy test: One of the most relevant yet overlooked email testing ideas is to always check your email campaigns’ overall quality and accuracy. This isn’t measurable in numbers, so do this test by sending your email to a few trusted team members. Then, have them check your email’s quality, accuracy, and other relevant feedback. It’s a simple test, but always effective and relevant.
Frequently Asked Questions (FAQs)
Some of the most important email A/B testing metrics are open rates, click rates, conversion rates, and bounce rates. These metrics are the basic indicators of the success or failure of your email campaign, and its overall effectiveness with your target audience. Open rates can measure the effectiveness of your subject line and preview text, while click and conversion rates measure your email content, and bounce rates indicate the health of your audience list.
The rule of thumb is that the longer an A/B email test runs, the more accurate its results. However, according to email experts at Mailchimp, the ideal testing period for open rates is two hours, while for click rates it’s one hour, and for testing revenue it’s 12 hours.
Yes, Mailchimp has tools for conducting email A/B tests. You can run A/B tests for your email’s subject line, sender name, sending time, and overall content and design. You can even test up to three versions of the same email. Mailchimp even lets you choose your own winning combination, or it can automatically determine one by itself.
Bottom Line
Email campaigns are a staple in marketing. And with so many possibilities, determining the right strategy is key to your success. That’s why email A/B testing is so important: it brings long-term guidance and value. With enough testing, you’ll eventually determine the best strategy for your business. Follow the steps and email testing ideas above to set up A/B tests for your email campaigns, and use the insights you gain to build better strategies.
Ready to get started? Try Mailchimp. All paid plans include subject line A/B testing, and the second tier and up have multivariate testing, giving you the most ways to get insights and improve your return on investment with email marketing. Check it out for free today.
