A/B testing

What A/B testing means

In A/B testing two versions of a page run at the same time and visitors are split between them at random. Afterwards the two are compared on which version reached the goal more often. The advantage over a before and after comparison lies in the simultaneity: weather, holidays, campaigns and seasonal peaks hit both groups, so they can no longer be used as an explanation.

How a test is set up

A test starts with a question, not with a design idea. The useful form is: if I place the shipping note on the product page, the number of abandoned checkouts falls. From that follows what is measured, namely the abandonment rate and not the time on page. Without that decision made up front, the number that looks better gets picked at the end.

The pitfalls that invalidate results

• Read too early: at the halfway mark one variant almost always leads. Stopping then is deciding on chance.
• Too few visitors: with little traffic a test runs for weeks or never returns a result. The required volume depends on the expected difference, not on patience.
• Several changes at once: anyone changing title, image and button together only learns that the bundle worked, not which part did.
• Wrong goal: a test on clicks can lift the click rate and lower the conversion rate.
• Uneven split: paid traffic must not concentrate on one variant.

Which tests are worth running

Not everything deserves a test. Large interventions in the buying path justify the effort, such as shipping notes, the number of fields in the checkout, the position of the buy button or price perception. Small changes to colours and wording return results that rarely repeat and are therefore of little value. With low traffic it is more sensible to adopt proven patterns than to run tests of your own.

What to do afterwards

The effect of a test is not permanent. Customers get used to new elements and the effect fades after a while. Anyone rolling out the winner should revisit the figure after a few weeks and build the next test on it. Testing is therefore an ongoing process and not a project with an end.

Frequent questions

• How many visitors do I need? It depends on the expected improvement. Small effects need a multiple of the traffic.
• Can it be done without a tool? In principle yes, with two shop versions and a trackable split, but in practice it is error prone.
• What if both are equal? Then the change is meaningless. That is a result too.