Every vendor in this category leads with a percentage. Conversion up by some amount, revenue per session up by another. We do not have one, and this is the page that explains why rather than burying it.
The reason is simple and not very flattering: we have not run a live experiment on a merchant's traffic. Without one, any figure we printed would either be someone else's store or an invention. Both are worthless to you and one of them is a lie.
Anyone quoting you an uplift before running a test on your traffic is quoting someone else's store.
What we would run instead
The absence of a number is not the absence of a plan. This is the experiment, and we would agree it with you before it starts rather than after it finishes.
- Eligibility: sessions reaching a category page in the pilot category
- Randomisation unit: the session, stable for its lifetime
- Arms: the engine on, against your existing order, held back
- Primary metric: agreed with you before the pilot starts, not selected afterwards from whatever moved
- Guardrails: p95 latency, error rate, fallback rate
- Stopping rule: fixed in advance
- Reported: as a comparison against the held-back arm, including when the result is inconclusive
The held-back arm is why the runtime ships with a holdback percentage on by default rather than as a setting somebody remembers to enable. A deployment without a control arm produces numbers nobody should act on, including any we would report to you.
What we can tell you today, and what needs a test on your traffic
our own scope, stated plainlyThe first two are properties of your catalogue and available immediately. The third is a property of your visitors and is not.
View as table
| Value | |
|---|---|
| Which fields to fix first | 3 measurable now |
| Whether the repair worked | 3 measurable now |
| Effect on revenue | 1 needs a live test |
What we can measure without an experiment
There is a whole class of claim that does not need traffic, because it is a property of your catalogue rather than of your visitors.
Whether your shelf can support the decisions buyers are trying to make is measurable the day you send an export. Whether a repair improved it is measurable the week you finish. Neither of those is a revenue claim and we are careful not to let them sound like one, but both are checkable immediately and both are actionable.
The cost of publishing this
It loses us conversations with buyers who need a vendor that already has the case studies, which is a legitimate thing to need and we would rather they find out now than in month four.
What it buys is that everything else on this site can be read at face value. The figures we do publish come from a catalogue we measured ourselves, they are labelled as ours, and the command that regenerates them is on the method page. That is a smaller claim than a conversion percentage and it has the advantage of being checkable.