This lesson will prepare you to:
- Explain how A/B testing helps determine which ads perform better.
- Use insights to choose the optimal recommendations for a future campaign.
- Use an A/B test report to identify actionable results.
Experiment with different creative versions
Learning which creative performs best can help you engage more people and inspire them to act. Using A/B testing allows advertisers to test different ad formats, visuals and copy to see which variations yield the best results.
How A/B testing works
In an A/B test, your audience is divided into random, non-overlapping groups. This randomization reduces the impact of outside factors and helps give your test the best chance of actionable results.
Each group sees ads or ad sets that are identical except for the variable you’re testing. Variables may include the audience, delivery optimization, placements, creative elements or product sets, among others. Note: We recommend testing only one variable. If you test more than one, it’s impossible to determine precisely what’s causing the difference.
Interpret the results
When you’re ready to interpret the results, your two key takeaways are the winning ad or ad set and the confidence percentage.
Winning ad set
If one ad set out-performs the other, it’s declared the winner. Winning ad sets or ads are determined by comparing the cost per result of each ad set or ad based on your campaign objective. We then simulate the performance of each variable tens of thousands of times to determine how often the winning outcome would have occurred.
We also provide a confidence percentage, which represents the likelihood you’d achieve the same results if you ran the test again.
For example, you run an A/B test using the creative variable, in this case a video against an image. Your results show the ad set with the video is the winning strategy with a 90% confidence percentage. This means the video ads had the lowest cost per result, and there’s a 90% chance that if you ran the same test again, you’d get the same result.
Take action to improve future campaigns
With these results, we recommend that you adopt the winning strategy. For any A/B test, a confidence percentage above 65% is an indication that using the winning ad set would likely achieve superior outcomes.
Once you’ve reviewed your results and understand which strategy performed best, you can:
Create a new ad from the winning ad set.
Create a new campaign based on what you learned.
You could also run more tests to refine your strategy further.
How does Eataly use testing?
A/B tests are ideal for data-curious advertisers. Let’s look at a success story from Eataly.
Eataly is a culinary marketplace founded in 2007 that aims to bring the best of Italian food and wine to the world. Recently, the company wanted to drive new traffic to the website and then encourage people to place orders.
The marketing team wanted to learn which ad format would most effectively drive consideration and conversions with new audiences, so they tested carousel ads against video ads. Their A/B test proved video delivered a clear advantage with the following results:1
2.4X return on ad spend from video ads
47% higher return on ad spend with video ads
51% higher conversion rate (click to website) with video ads
33% lower cost per action (landing page view) overall
Further, the overall campaign showed the Eataly team the value of scaling its video assets for future campaigns.
A/B Test to compare platforms
SHEIN is a global ecommerce retailer specializing in fast fashion. When its marketing team was crafting its campaign strategy, they wanted to determine the impact placing ads in Instagram Stories might have, so they used an A/B test with two different video ad campaigns. One used Facebook and Instagram Feeds and the other used ads in Instagram Stories in addition to the Feed placement used in the first.
The video suggested items of clothing for different environments where busy young women may find themselves during a typical week. To capture attention, it included upbeat music, bright colors, pop-up text, emojis and geotags as well.
After comparing the two campaigns, the team found that including ads in Instagram Stories doubled their reach. They also recorded a 9% increase in purchases at an 8% lower average cost per purchase.2
This test proved the value of Instagram Stories for building brand awareness and growing conversions. Based on these results, the SHEIN team decided to continue to incorporate Instagram Stories in their future campaign efforts.
Check out these resources to continue developing your skills.