How to A/B test a free shipping bar on Shopify
Split your traffic between two placements of the bar, or against a group that sees no bar, and compare revenue per visitor and orders. A/B testing requires the Optimize plan.
What you can test
An A/B test compares placements of your progress bar on real traffic. Each version is a placement you already set up in Theme integration, edited with the usual placement editor. For example:
- a top-of-page banner against a bar inside the cart drawer;
- the same placement with two designs: duplicate the placement, then switch its design;
- your bar against no bar at all, with a holdout group.
A visitor keeps the same version from page to page, and from device to device when signed in to their customer account. The draw happens in the browser, from the configuration already loaded: tests do not slow your store down.
Who can use it
A/B testing is part of the Optimize plan. The A/B testing (Beta) menu entry is visible on every plan; without Optimize, the page explains the feature and links to the plans. See subscription and the pricing page.
Create a test
- Open A/B testing (Beta) and click Create test.
- Control integration: pick the placement shown to the control group (A).
- Variants: add the other placements to compare. Click Add a holdout (no bar) to measure the bar against no bar. The holdout group loses what the bar would have earned, so keep it small and the test short.
- Edit variants: adjust each placement with Edit placement if needed.
- Configure the test: enter a Test name, a Start date and an End date, the Primary metric (Revenue per visitor, Conversion rate, Average order value or Add-to-cart rate) and the Traffic split, which must total 100%.
- Click Save as draft or Launch test.
A placement can only be in one active test over a given period. If it is already used, pick another placement or schedule the test for another window.
A test is Draft, Scheduled, Running, Paused or Completed. From the test page you can Start, Pause, Complete or Delete it.
Read the results
The Results tab shows one row per variant: Visitors, Add-to-cart, Revenue, Revenue / visitor, Conversion, Uplift / visitor, Significance and Time to goal.
- Revenue per visitor is the fairest comparison when variants get uneven traffic.
- With a holdout, uplift is measured against the group that sees no bar. Without one, it is measured against the control.
- Needs more data means it is too early to conclude. Wait instead of reading a gap that could be chance.
- A sample ratio mismatch warning means the observed split differs from the one you set, often because visitors browse in private mode. Read the results with caution.
The Orders per variant card counts the full orders placed by visitors of each variant, holdout included: Orders, Average order value, Reached a goal and the average order value against the holdout or the control. It stays empty while the test is young, or if the app cannot read your orders.
Show a specific version with a link
Under Force a variant by link, Copy link gives one link per variant. Use it to check a version yourself, or in an email campaign or an ad. Whoever opens it sees that version and stays on it. These visits are counted separately, so they do not bias the comparison.
After the test
When a test is paused or completed, the bar is no longer split between variants: every placement of the test shows at the same time. Turn off the losing placement in Theme integration to keep only the winner. If your store suddenly shows two bars, check your tests first. See theme placement.
The home page of the app lists your Running A/B tests.
Something missing? Write to [email protected], or try it first in the demo.