Offer testing Evidence
Two variants, one winner, called on deposits
Almost every SMS test we inherit is measured on clicks, which is why almost every one of them is wrong. A bonus that pulls a big click rate and a small deposit rate is a worse offer that looks like a better one. We split on the segment, hold everything else still, and let the deposit column decide.
1 in 3
Tests that overturn the favourite
6 hrs
Median time to a call
9,000
Minimum arm size we will call on
How a job actually runs
Frame
We name the one thing being tested and what result would change your mind.
Split
The segment is halved on a hash of the player id, so the halves are comparable.
Watch
Deposits are counted live against each arm as the waves go out.
Roll
The winner takes the remainder of the list, usually the same evening.
What is in it
Everything below is included at the standard engagement. Nothing here is an upsell later on.
- Matched halves drawn from one segment, not two similar ones
- One variable changed at a time, so the result means something
- The deciding metric agreed in writing before the send
- Minimum sample enforced, so a lucky evening cannot call a test
- Losing variants stopped mid-flight rather than left to finish
- A running library of what has already been proven on your list
Get offer testing priced
Tell us the market, the segment and the offer. You get the plan and the number back, not a calendar invite.