An Ultimate Guide to A/B Testing Product Pages

Marketing has evolved over the years as digitalization continues to accelerate. Some might say that traditional marketing was much simpler and required less work. On the plus side, however, the digital age has paved opportunities to connect with our consumers more.

Unlike traditional marketing, digital media allows us to edit a pice of content anytime we want. Thus, A/B Testing has become one of the most important processes to improve conversion rates by discovering the optimal user experience for customers.

So, why is this an important part of marketing? How exactly do you conduct an effective A/B Test to optimize your product pages?

Why Test?

A/B Testing (sometimes known as ‘split testing’) is an experiment that compares two or more versions of a digital asset, such as a web page or mobile app, to reveal which version has the best outcome that scales conversion rates and hits other metrics.

Thus, it allows teams to enhance their pages’ user experience and content while obtaining data based on the experiment’s results.

A/B tests provide insights that reveal how certain variables affect customer experience. When done right, they enable you to increase user engagement and sales, reduce bounce rates, and optimize customer-centric content.

Let’s dive deeper into how A/B testing is done right.

How to Conduct A/B Tests for Product Pages

1. Research & Choose a Product Page to Start With

Before beginning the A/B Test, you must have enough data to know how your product page is performing (if your product pages aren't doing so well, check out here useful tips on how to build high converting mobile product pages). The important metrics to keep in mind include clicks, checkouts, traffic, click-through rate, users, cart rate, and bounce rates. These will measure the performance of your experiment.

It’s important to choose which product page to optimize first. Usually, you’d want to conduct A/B Test on pages that yield you the highest revenue. Hence, this means the products that make the most sales.

Alternatively, you may also use A/B Testing to troubleshoot high-demand products that aren’t performing well. Perhaps tweaking the user experience may get you that higher conversion rate you’re missing out on.

It’s also important to include a product page that will serve as the “control group” – the one that won’t be undergoing any changes. To measure whether changing a certain variable improves or worsens user experience and conversion, you must be able to compare it in real-time with the same environment. This will represent the baseline of effectivity and current conversion rate, enabling you to gauge if certain changes are needed. It may also highlight what’s already working for you.

2. Set Up the Environment

According to multiple web optimizers like Rich Page, you need at least 5,000 unique visitors per week to test a page. Of course, the more web visitors you have, the more accurate the results would be.

Therefore, you must determine the right traffic allocation to get the best results. Here are two ways to allocate traffic across your test pages:

• Manual allocation:

  • Traffic will be evenly distributed among variations. Once the test provides significant results, the best variation will be directed to all visitors. In this test, only the best-performing variation will be chosen throughout.

  • This is a good practice when you’re going to be doing it for the long run, which requires a statistically significant result.

• Dynamic allocation

  • The best-performing variation will be gradually assigned to a larger portion of visitors as more data are gathered. Over time, traffic will be routed to the best-performing variation based on real-time data. Thus, the best variation shown may significantly change over time, depending on real-time performance.

  • This is optimal for testing campaigns with a time limit, such as holiday sales in which the variable is a hero banner. This requires more aggressive work but can be highly effective.

How do you determine when there is a statistically significant difference? Scientists agree that a 5% change is significant enough to attribute it to a change in variable, and not just to chance. This is an industry-standard that is often seen as a reasonable risk.

This means that a change in variable can be attributed by an A/B test that receives a statistically significant result (e.g. a 5% or higher change in traffic).

Lastly, set a timeline for the experiment. Ideally, it should run for at least two weeks. But as mentioned, it would depend on the purpose of your A/B test.

If you’re testing a timeless product page, then it would be best to test until you have enough significant data – however long that may take. If you’re testing to optimize your holiday sales campaign, then the timeline should be set accordingly.

3. Create Variations Based on User Behavior Data

Determine which variable you want to change and test. Use your data and expertise to determine which variable needs improving. You may do a competitor analysis, and consult your SEO team and design team when changing a variable.

Here are certain variables you can consider:

• Copy

• Design and layout

• Navigation

• Forms

• Call-to-action

• Reviews

• Content structure

• Product images

A/B testing is different from multivariate testing, which more or less has the same mechanics, except it compares more variables at once to reveal how these interact with one another. However, you need a substantial amount of traffic to effectively conduct this, otherwise too many changing elements at once add up to a large number of combinations.

Thus it is often recommended to begin with changing one variable per test to gain clearer insights. This already enables you to see which variable in particular affects user experience. So, it’s best to design multiple well-designed A/B tests instead of investing more time in running a multivariate test.

This doesn’t mean that you should perform only one test at a time. You may change another variable on a different test page.

For example, you may change the copy format in test A, change the product page layout in test B, and change the call to action placement in test C. This is what we mean by variation.

4. Analyze Results with Heatmaps and Other A/B Testing Tools

Data analysis is a crucial part of getting the best outcome from each test. Your test is only as good as your data interpretation. Each engagement with each variation should be painstakingly measured to determine which set of variables produced a successful outcome.

So, it’s best to determine which metrics you value the most, and which tool is best for your priorities.

Here are certain variables you can consider:

• Omniconvert

• Google

• Oracle Maxymizer

• ContentSquare

• Optimizely

• VWO

How to Optimize Your Product Pages

A/B Testing should not just be a one-off experiment, but a continuous effort to optimize your pages in every possible way.

So, what exactly do you do with your data to optimize your product pages?

Validate Your Data

Just because you have a winning test, it doesn’t mean you should hurry and implement all the changes throughout your website. You must first validate your results such as testing the winning variation against the control group to see if it performs well the second time around.

An alternative worth trying is to allocate 100% of the traffic to the winning variation again, and see how it performs overall.

Potentially Mix Variations

Some A/B tests don’t reveal a clear-cut winner. Sometimes, multiple variations perform just as well. You may either go with the best-performing variation or combine certain variations into one user experience (for which you could then conduct multivariate testing).

This will all depend on certain data factors such as the type of visitor (whether it’s organic traffic or paid), traffic source, device type, and such. Carefully analyze the user behavior on each winning product page, and decide from there.

Recognize the Positives of The Control Group

Having losing variations does not mean you wasted your time. A/B Tests show you what works and doesn’t work – which is a step closer to optimizing your product page.

Your results may show which part of the control group you have been doing right (therefore, should not change at all), and which part needs optimizing. There are many ways to skin a cat, and A/B Testing just shows you which is optimal for your product.

Use the AIDA Model

Every eCommerce marketer should know about the AIDA model, which takes you through each stage of the user journey: awareness, interest, desire, and action.

When creating or optimizing a product page, much more an entire eCommerce marketplace, you should take into consideration which part of the user journey your customer is in. A product page’s visitor must either be at the awareness or interest stage.

When your web visitor clicks through the product details or customer reviews, they are definitely at the interest stage. When they add the product to the cart, they are at the desire stage. Lastly, they are in the action stage when they reach the checkout process.

Therefore, you must consider each stage of the user journey by making sure the changed variable fits according to the AIDA stage your customer is in.

Case Study: The Success Story of Zalora’s A/B Testing

Zalora, one of the biggest online fashion stores in Southeast Asia, saw a 12.3% increase in their checkout rate after optimizing their product pages.

According to VWO’s success story, Zalora’s new product page design was able to highlight alluring features such as their free deliveries and return policies since data from their customer service team initially revealed that their users weren’t aware of these aforementioned features.

Therefore, VWO ran A/B tests for Zalora with variations that point these features out. Here’s what the control group looked like:

And here are the different variations:

In the end, the first variation showed more success over the control group and the other variation since the CTAs are clearer and more cohesive.

In this example, you can clearly see the beauty of A/B Testing – how a simple change can yield such a huge gain.

Conclusion

A/B testing is one of the most powerful ways to improve conversion rates and increase sales. It is a continuous effort to consistently improve your website’s user experience.

Although, there is a proper way to do A/B testing, don’t be afraid to fail. Experiments, in nature, have the tendency to fail, which is the beauty of it. The more you test, the more data you get. A failed experiment does not conclude failure – in fact, it means you’re a step closer to scaling your eCommerce business.

That said, we do hope our guide can help you get the most out of your experiments and claim those wins.

Looking for additional e-commerce related tips? Check out our article about ecommerce personalization.