Reading A/B test results, picking a winner and the automatic winner
Open a test from A/B tests, or tap its A/B pill, to see its results. They are drawn as lines flowing from what you test to each version, thicker for the versions that get more visitors, with the leader glowing. Each version has a card with its funnel: visitors → orders → confirmed → delivered. Orders still with the courier are shown as "with courier" and are not counted as delivered or returned yet. An order counts only if it contains the tested product.
The goal decides the winner. You choose it when you create the test:
- Delivered profit per visitor (recommended): what you really earn after delivery, returns and costs. It uses the same costs as Analytics.
- Delivered orders per visitor.
- Orders placed per visitor: the fastest result, but it ignores refused and returned parcels.
Chance to win is the probability that a version is the best for the goal. A version with the most orders is not always the winner: the page warns you when one version gets more orders but another gets more deliveries.
When a winner can be called: at least 7 days, at least 200 visitors per version and 30 finished orders (delivered or returned), and one version with a 95% chance to win. Until then the page says Too early to call and why, with an estimate of the days left.
Pick the winner automatically: a switch on the test page and in the last step of New test. You can turn it on or off at any time, even in the middle of a test. When the rule above is met, the test ends by itself within the hour, the winning version is kept and you get a notification.
Send more visitors to the winner before the test ends: the Visitors split box on a running or paused test.
- Drag the white handles on the bar (or tap the bar, or use the arrow keys), or type a version's share: the other versions adjust so the total stays 100% (with two versions, typing 80 for A makes B 20). Even split appears while the split isn't even, and the back arrow in the card's corner undoes your changes. Then Save split. Each version keeps between 1% and 99%.
- Shift visitors to the leader automatically: every hour, once each version has 100 visitors and the test has run a day, the version ahead gets a bigger share. Every version keeps at least 10% so the test keeps learning. Saving a split by hand turns this off.
- People already in a version keep it; only new visitors follow the new split.
Ending a test yourself:
- Keep version X ends the test with that version:
- Images, title or description: the winning version replaces the product's own content. Keeping version A changes nothing.
- Theme: the winning theme becomes your published theme.
- Pages: by default, every link of the test keeps opening the winning page, so ads that point to a losing page still land on the winner. Turn this off in the confirmation, or later with Stop sending links to the winner.
- End test stops it without a winner. Everyone goes back to the original.
- Pause shows everyone version A until you resume.
- Results keep updating after a test ends, as its orders get delivered or returned.
- An ended test can't be restarted. Create a new one.
Preview (on each version, while the test runs) opens your store showing that version, without counting you as a visitor.
On an order: the order window shows which A/B test version the customer saw.