A/B Testing in Holiday and Peak Seasons – CRO Tips

By Greg Thibault, Organic Search and Conversion
TOPICSConversion, Featured

A/B Testing During Holiday and Peak Seasons: Conversion Rate Optimization (CRO) Tips

As we all know, retail sales experience a significant boost during the holiday season. But according to Deloitte’s 2020 Holiday Retail Survey, this year is going to further skew toward online versus in-store shopping.

How will this significant increase in your website’s traffic affect CRO and A/B Testing?

You may be thinking: great! This is the highest traffic we’ll get all year, so we should test heavily to take advantage. Think of all that we can learn during this time!

But here’s the problem: Users are much more shopping-motivated during the holidays than at other times of the year. So they are unlikely to be as influenced by traditional A/B testing changes in user experience (UX). People will put up with — and work through — the worst UI imaginable in order to make a purchase during the holiday season (especially if there are discounts involved). That means the data we receive from these sorts of tests during holiday/peak periods isn’t representative of “normal” user behavior.

Pitfalls of Holiday A/B Testing

Thus, we can’t ascribe increases in conversion rates to anything we’ve done to the user experience. This is an illustration of sample pollution, which means the sample population was affected by external factors outside of our control. This causes us to question what results we may see from typical conversion tests.

Here is the bottom line: During your holiday or peak seasons, you should stay away from testing changes that are UX or usability-focused (e.g. site-wide navigation, product page hierarchy, etc.).

Best Conversion Tests for Holiday Sales

But are there any types of conversion tests we can run during this time that are likely to provide us with actionable results? You bet! The two biggest differentiators for websites during this time tend to be around their promotion strategy and conveying trustworthiness to potential customers.

With that in mind, here are the best things to test during your peak/holiday season:

  • Ideas for Black Friday, Cyber Monday and December promotion tests:
    • Free shipping thresholds (at $50/$75/$100)
    • 10% off vs. 15% off site-wide
    • Dollar thresholds for discounts
      • Spend $50 to get 10% off, spend $100 to get 20% off, etc.
  • Elements that remove fear, uncertainty and doubt. Think about how you can provide that final nudge to entice a customer to purchase. How do we provide assurance that the user is making the right decision in buying from us?
    • Messaging Tests
      • Value props
      • Social proofing
      • Return policy
      • Shipping guarantee
      • Urgency to purchase (info on when promo ends, deadline for shipping, etc.)
      • Wording and location of messaging

Although users are strongly motivated to make a purchase and will work their way through even the clunkiest UI to purchase the products they want, you can still successfully execute A/B tests to improve your conversion rate during peak seasons. You can stand out from the pack by giving your users the right promotional offers and effectively communicating your trustworthiness, and you can accomplish this with well-planned CRO A/B testing.