Segmenting Test Results – Conversion Wednesday

By Chris C.


Segments are the different groups of customers that use your website. While each customer experience is unique to that user alone, there are parts of their experience that are the same as other customers.

Some examples would be:

  • New vs. Return Visitors
  • Weekend vs. Weekday Visitors
  • SEO vs. Direct referral traffic

So, lets run through an example of how segments can alter the conclusions you make about the success of your testing. The following test data is actual test data pulled from a live test.

The overall numbers are pretty bland. No real change in conversion rate or revenue. A small lift in pages viewed per visit. It’s pretty disappointing test results when viewed at the top level. These kinds of numbers can deflate your excitement about testing.

However, when we break this data up by weekday visitors vs. weekend visitors segments, the results suddenly become valuable. Just look at the difference! Obviously this test is working for weekday visitors. Almost a 7% increase in conversion rate and almost a 5% increase in overall revenue?!?! Those are big results when annualized… And look at the weekend numbers! It’s the exact opposite!

If you didn’t view your test data by breaking out into segments, you might have made a change to your site that decreased weekend conversion rate by almost 16%!!

Considering I know that Time Per Visit is up 4.3% on weekend vs. weekday traffic, we can see that this test is helping customers find what they need faster during the week. However, it’s encouraging browsing behavior on the weekend! So, rather than make this change to the entire site, we could make this change so only weekday visitors see the test. This only gives you the benefits and removes the negative affects!

Plus, these segments give you an insight to your customer’s behavior on weekdays vs. weekends. It’s obvious your weekend visitors are lounging around and browsing your site with no intent to buy! They are spending more time on your site, consuming more pages, but buying less. This gives you a target for follow up testing with your weekend traffic: how to you turn customers from browsing to buying mode? The answers to this question could be your next test!

Remember to segment your test data. When viewing your test data for all visitors included in the test, you might not be seeing the full picture. The results on the previous slide were just for weekday vs. weekend visitors. That’s an easy one! What about new visitors during the week vs. return visitors on the weekend?!?

Make sure to segment your data into different groups to help get a better idea of how your customers are using your site! This ensures you are only making positive changes to your site and it helps you learn more about your customers.