Importance of Your Control – Conversion Wednesday

By Exclusive Concepts Blog Team


Last time I went through how to design an A/B test and a multivariate test. I also explained how your site traffic levels would determine which test to use. The more variables in your test, more traffic will be required. This means more time needed to confidently identify test winners. Because testing different variables at the same time can require a lot of traffic, sometimes shortcuts are presented as options.

Today I’m going to go over the importance of your control and two ways to ensure it is included to so you don’t try shortcuts that will only deliver more headaches. Before we get into what to avoid, first we have to know what we’re looking for. During a test, a “control” is the version of your website or webpage as it stands before you make any changes to it.

In order to have a control version of your webpage, you first need a controlled test. Marketing efforts, Pay Per Click campaigns, or a holiday season could all cause conversion rates to improve (or drop) regardless of the test you are running. Having a controlled test helps to minimize these outside effects. Any popular testing software on the market will provide you with a controlled testing environment. To make this easier, I’ll assume you have testing software and you have can set up a controlled test environment.

We’ll use the button example again. Let’s say you currently have a green Add to Cart button on your product pages and you want to test how a red one would perform across the entire site, you’ll have two versions to test:

  1. Recipe A: your control (or green button), and
  2. Recipe B: or the new red button

When testing the red button, it’s important you don’t change anything about the green button otherwise you won’t know if the site was performing better before you made changes! For confident test results, you need a control in order to have something to compare against. So, here are two common mistakes to avoid to make sure you always have a control running in your test:

First, for confident results, avoid using the “before and after” approach. This approach implies that you’ll make the change live on the site without testing and simply look at “before and after” data in your analytics. This approach is not a controlled test environment and removes the entire word “control” from the test. This approach often causes more questions than answers.

As an example: If you have a product on your site that suddenly shows up on the first page of Digg and your page gets thousands of additional visits for the next few days as a result, how do you compare that with a previous date range? Your data is sure to be skewed by that additional traffic.

Avoiding the “before and after” approach ensures you are including a control version of your site and testing your variations against that control at the same time. This way, if one of the previous examples happens, it happens to all variations at the same time. This way you can be confident that the changes you see were unaffected by that outside variable.

The second mistake to avoid would be to change your control because you assume something else will perform better. Just like the first example, this skips the whole testing process! Don’t assume that the boring, square, green button will perform worse than the fancy rounded, shiny style you want to test. If you change that green button to the rounded, shiny style and then test that against a rounded, shiny, red button, how do you know that either of those two buttons actually performed better or worse than the original, boring, square, green one? You can’t. There’s another question that starts to pop up when you do this: exactly how much did the new button vs. the old button contribute to or take away from my overall revenue?

Don’t assume that what is currently on your site isn’t working or won’t work better than a new idea.

The bottom line is: make sure you have a control version of your webpage while testing! It seems a little silly to have to point this out, but don’t let your concern for saving time get in the way of delivering measurable results. Without a control, you won’t be able to:

  1. confidently know what site changes are helping
  2. measure what variables have the largest affects, or
  3. measure how much these site changes are affecting your revenue.

Having a control to test against ensures that the changes you are making are helping your site and you’ll be able to show how much each change is helping.

Exclusive Concepts offers a full service testing option for our clients called Conversion Booster that will do the work for you in gathering these analytics and determining the best changes for your site to maximize conversion rate. Then test those changes in an AB or Multivariate test. Give us a call at 1-800-504-4324 if you’re interested in learning more.

We would be honored to perform a free conversion audit on your qualified website today.