Become an affiliateStart your program

Technology and tracking

Deduplication (dedup)

Also called dedup, de-duplication, double counting prevention, duplicate check, conversion deduplication.

Short answer

Deduplication is the check that ensures the same order only triggers commission once. The order’s ID is stored together with the program, and a repeated ID is rejected instead of creating a new conversion.

Definition of Deduplication (dedup).

Duplicates arise for entirely ordinary reasons. The customer refreshes the receipt page, the browser resubmits the form, the phone loses its connection mid-request and tries again, or both a tracking pixel and a webhook report the same order. None of this is fraud – it is the normal state of an internet with retries built in everywhere.

The solution is a unique key per event. The key consists of the order or invoice ID combined with the program, and it is enforced as a constraint in the database rather than as a check in the application code. The difference is crucial: two requests that hit the server in the same millisecond will both see “the order does not exist yet” if the check lives in the code. A unique constraint in the database can only be won once.

The word covers two things that are often confused. Technical dedup is about the same event being reported several times and only being allowed to count once. Channel dedup is about two marketing channels each claiming the same sale – the affiliate link and the ad the customer also clicked. The first is data hygiene, the second an attribution question decided by the advertiser’s rules.

The key decides everything

A good dedup key is something the store’s own system has already assigned and never reuses: the order number, the invoice ID, the payment’s ID at the payment provider. A bad key looks unique without being so. The customer’s email plus the amount fails the first time someone orders the same thing again, and a timestamp fails when two orders fall in the same second.

The key must also be determined by the store, not by the browser. If it is sent from the client without being tied to something the server can recognise, the same order can be submitted with a new ID and pass the check.

Subscriptions must not be deduplicated away

With recurring commission, each renewal is a new, valid event with its own commission. If you use the subscription’s ID as the dedup key, only the first payment counts, and every renewal is rejected as a duplicate – an error that is hard to spot, because the system behaves exactly as it was told to.

The key must therefore contain the invoice rather than the subscription: the invoice ID for the individual payment is already unique at the payment provider.

How it goes wrong

  • 01

    The check only lives in the application code

    A check for “does the order already exist” followed by an insert is two actions with a gap between them. Two simultaneous requests can both get through the gap. Only a unique constraint in the database closes it.

  • 02

    The key is not unique enough

    Composite keys made of customer details and amount look unique until the same customer orders the same product again. Then a genuine sale is rejected as a duplicate, and the affiliate loses a commission without being able to see why.

  • 03

    Renewals mistaken for duplicates

    Subscription renewals resemble earlier payments in amount, customer and program. If the key is tied to the subscription rather than the invoice, all recurring commission disappears after the first month.

At Affilyflow

Deduplication (dedup) in practice

At Affilyflow, conversions are deduplicated on order or invoice ID per program. The same order can therefore never count twice, whether it comes in through the tracking script, the Shopify app’s pixel, the WooCommerce plugin or a Stripe webhook.

A store’s webhook can only credit the store’s own program – if an event points to another program, it is rejected. For subscriptions, the key is tied to the individual invoice, so renewals each count separately.

Every single tracking event is logged – including the rejected ones. A duplicate does not disappear silently: it stays in the event log, so it can be verified afterwards.

Frequently asked questions

What does deduplication mean in affiliate marketing?

Deduplication is the check that prevents the same order from triggering commission more than once. The order’s ID is stored together with the program, and if the same order is reported again, it is rejected rather than recorded as a new conversion.

Why does the same order get reported twice?

Because retries are built in everywhere. The customer refreshes the receipt page, the browser resubmits the form, the connection drops mid-request, or both a tracking pixel and a webhook report the order from their own side. This is normal behaviour and not a sign of fraud.

How do you avoid paying commission twice for the same sale?

By tying each conversion to a unique key that the store’s own system has assigned – typically the order number or invoice ID – and enforcing it in the database rather than in the code. A check in the code can be bypassed by two requests arriving at the same time; a unique constraint cannot.

Do subscription renewals get deduplicated away?

They must not. Each renewal is a separate payment with its own commission, so the dedup key must contain the invoice rather than the subscription’s ID. If the key is tied to the subscription, only the first payment counts.

Read on