Become an affiliateStart your program

Measurement and metrics

A/B test (split test)

Also called A/B testing, split test, split testing, A/B/n testing.

Short answer

An A/B test is a method where two versions of the same page, link or placement are each shown to half of the traffic over the same period. Any difference in conversions can then be attributed to the change, not to swings in traffic.

Definition of A/B test (split test).

An A/B test is also called a split test. Version A is the current one – the control – and version B contains one change. Traffic is split randomly between the two, both run at the same time, and the same success metric is counted for each group. Running them simultaneously is the whole point: if A runs in March and B in April, you are measuring the season as much as the change.

In affiliate marketing, four things are typically tested: where on the page the link sits, what the anchor text says, whether the recommendation is a text link or a banner, and which landing page the link points to. Each test should isolate one change. Change both placement and wording at once, and afterwards you only know that something worked – not what.

The result is only usable when the number of conversions is large enough to rule out chance. With 20 conversions in each group, a 15 % difference can be pure luck. The smaller the difference, the more conversions it takes: to tell a few percentage points apart from noise, each variant needs conversions in the hundreds. The test period is measured in whole weeks, so weekdays and weekends weigh equally in both groups.

How to set up a test you can actually read

Decide three things before the test starts: the hypothesis, the target metric and the end date. The hypothesis must be falsifiable – a link above the fold gets more clicks than a link below the product description – and the target metric must be the thing you actually want more of. If you measure clicks, you can win more clicks and lose money, because those clicks convert worse.

Record the variant, period, clicks and conversions for both groups as you go. Without those notes you cannot later separate the effect of the change from a week when the store happened to run a campaign.

What you cannot split test

Everything outside your own pages is fixed: the commission rate, the attribution window and the store's checkout belong to the program, not to the placement. You can test how you send the traffic on its way – not what happens after it lands.

Traffic volume also sets a limit. A page with 200 visits a month and 4 conversions cannot detect a 10 % difference. There, a test is not the answer; a bigger change is.

A/B test vs. multivariate test

AspectA/B testMultivariate test
Number of changesone at a timeseveral elements combined
Answersdid this one change workwhich combination works best
Traffic neededtwo groups to fillevery combination to fill – grows fast
Suited topages with ordinary trafficpages with very high traffic

Choose an A/B test unless the page has enough traffic to fill every combination in a multivariate test – which very few affiliate pages do.

How it goes wrong

  • 01

    Stopping the test when it looks right

    If you check daily and stop the first time B is ahead, you will systematically stop on random fluctuations. The end date must be fixed before the test begins.

  • 02

    Two changes in the same variant

    A new placement and new anchor text in version B give a result you cannot act on. Roll out one change, measure, then test the next.

  • 03

    The winner never gets rolled out

    A test with no decision afterwards is just lost traffic. Agree in advance which result triggers which action.

At Affilyflow

A/B test (split test) in practice

Affilyflow does not run split tests for you, but it provides the data a test has to be read from. Every click is created as its own row with the program's rate and attribution window copied in, and every tracking event is logged – including those that did not become a sale. If you can tell which variant sent which click, both groups can be counted from the same source.

Frequently asked questions

How long should an A/B test run?

Run the test in whole weeks, so weekdays and weekends weigh equally in both groups, and set the end date before you start. The period must be long enough to collect enough conversions per variant – but not so long that the store's campaigns or the season have time to change along the way.

How many conversions does an A/B test need?

It depends on how small a difference you want to detect. With 20 conversions in each group, a 15 % difference can be chance alone; to tell a few percentage points apart from noise, each variant needs conversions in the hundreds. If the page does not have enough traffic, one large change measured over a longer period tells you more than a split test of a small one.

What is the difference between an A/B test and a split test?

There is no difference. Split test is another name for the same thing: traffic is split between two versions running at the same time. The term A/B/n test is used when there are more than two versions.

Can you A/B test affiliate links?

Yes, everything on your own page can be tested: placement, anchor text, format and which landing page the link points to. What sits with the store – commission rate, attribution window and checkout – is part of the program and cannot be varied per visitor.

Read on