How to A/B test your free shipping bar on Shopify

What the test measures

An A/B test answers one question: how much does the progress bar earn you? Each visitor is randomly placed in one of two groups. The first group sees your store as it is today, with the bar and your tiers. The second group shops without the bar. At the end, Yaco compares the revenue per visitor of the two groups.

You choose what the second group loses. With The bar and its offers, it gets neither the bar nor your gifts and Yaco discounts: the test measures everything the bar earns you. With Only the bar, it keeps its gifts and discounts: the test measures what showing the bar adds on top of your offers.

A visitor stays in the same group from page to page and on later visits, and from device to device when signed in to their customer account. The draw happens in the browser, from the configuration already loaded: the test does not slow your store down.

What visitors without the bar get

In both cases:

• No progress bar anywhere: not in the cart drawer, on the cart page, on product pages, in app blocks or at checkout.
• Product recommendations still show, without any mention of tiers.
• Offers that don't come from Yaco stay as they are, such as your shipping rates, your own automatic discounts and discount codes.

With The bar and its offers (the default):

• No tier is unlocked: no free gift is added, the gift chooser never opens, and no Yaco discount or free shipping applies, on the store or at checkout. A gift already in their cart is removed.

With Only the bar:

• Tiers unlock as usual: free gifts are added, the gift chooser opens when there is a choice, and Yaco discounts and free shipping apply, on the store and at checkout. Only the bar, the offers summary and tier mentions in recommendations are hidden: the gift chooser, the gift button and the gift list stay, so visitors can still claim their gift.

Only your published theme is tested. The theme editor and theme previews always show the bar. Visitors whose browser blocks storage, as some private modes do, are left out of the test: they see the bar and are not counted.

Who can use it

A/B testing is included in every paid plan: Starter, Premium, Optimize, and the older plans that are no longer offered. The free plan does not include it. The A/B testing (Beta) menu entry is visible on every plan; on the free plan, the page explains the feature and links to the plans. If your store moves to the free plan or uninstalls the app while a test runs, the test stops. See subscription and the pricing page.

Launch a test

1. Open A/B testing (Beta).
2. Under Pick a setting, click 50/50, 70/30, 80/20 or 90/10. Each one shows its recommended duration for your store, and one click sets both the split and the duration. 50/50 is marked Fastest. To set the split yourself, move the Split of visitors slider, from 1% to 99% of visitors with the bar, one percent at a time.
3. Under Difference to detect, keep 15% (standard) or pick 20% (large difference) or 10% (fine difference). The test concludes if the bar changes revenue per visitor by at least this much.
4. Under What visitors without the bar lose, keep The bar and its offers or pick Only the bar.
5. Under Duration, keep the recommended duration or pick another one, from 1 to 8 weeks.
6. Click Launch the test. A window sums up the test before it starts: its duration, its end date, the share of visitors who won't get the bar and what they lose. Confirm to launch it.

Only one test runs at a time. The split, the difference to detect, the duration and what visitors without the bar lose can't be changed while the test runs: stop it and launch a new one. You get an email when the test starts, with a link to the test.

How the duration is recommended

Yaco reads your online orders of the last 28 days: how many you get per day, and how much their amounts vary. From there, it works out how many orders the test needs to detect the difference you chose in revenue per visitor with the usual standards (95% confidence, 80% power), and turns that into weeks.

• The duration is rounded up to whole weeks, so every day of the week weighs the same: from 1 to 8 weeks.
• The finer the difference to detect, the more orders the test needs, so the longer it runs. Halving the difference takes four times as many orders.
• A split other than 50/50 needs more orders, so the recommended duration grows. Near the extremes, such as 95/5, it quickly goes past 8 weeks.
• 8 weeks is the maximum. When your store would need more, the test is set to 8 weeks and the form tells you the smallest difference it can still detect.
• With too few recent orders, or when Yaco can't read them, the duration is set to the maximum.

The ⓘ icon next to each recommended duration shows the calculation: the orders needed, divided by the orders your store gets per day, gives a number of days, rounded up to whole weeks.

If you pick a shorter duration than recommended, the form tells you the smallest difference the test will still be able to see.

Follow the test

The A/B testing (Beta) page shows the test in progress: the day, the revenue per visitor of each group, the gap between them, the probability that the bar earns more, and the orders collected out of those needed. The number of orders needed is set at launch and doesn't change during the test. A table adds Average order value, Conversion rate, Orders, Visitors, Revenue and Orders above a goal threshold for each group.

The figures move a lot in the first days. Don't stop the test as soon as a gap shows up: it is the most common way to get a false result. Once the orders needed are collected and the test has run at least 2 weeks (or its full planned week), the page shows Sample reached: you can then stop the test without losing its verdict. Otherwise, it runs until its planned end.

The home page of the app shows a short version of the test in progress. While it runs, the other figures of the home page include both groups.

Your team is never drawn into a group: a browser or customer account left out of the stats, by hand or automatically, always sees the bar, and its visits and orders don't count (see statistics). If you never land in the group without the bar, this is why.

Under Check both versions, two links open your store in either group, so you can see what your visitors get. They are the only way for your team to see the store without the bar. Visits through these links count in their group, except your team's: tick Count my team's visits to count the visits and orders it makes through them.

Stop a test

Click Stop the test, then confirm. All your visitors see the bar again. You can stop a test at any time. With Sample reached, it keeps its verdict. Before that, its result is only a trend. The confirmation tells you which case applies.

Read the result

The test stops by itself at the end of its planned duration, and you get a recap email. You get it too when you stop the test yourself. The page then shows:

• the verdict: The bar earns you more, Your store did better without the bar, No measurable difference, Not enough orders to conclude or, for a test stopped before its end date without the orders needed, Test stopped early: no verdict;
• the revenue per visitor difference in large type: green when the bar wins, red when your store did better without it;
• whether the duration was sufficient: at least 2 weeks (or the full planned week for a 1-week test), and either the number of orders needed or the planned end with a significant difference;
• the revenue per visitor of each group, and the full table.

How the comparison works:

• Revenue per visitor is the order subtotal of a group divided by its visitors. It counts both effects of the bar: more orders and bigger carts.
• Orders above a ceiling count at the ceiling, so that one exceptional order can't decide the test. The ceiling is the 99th percentile of your order amounts, set when the test is launched.
• A difference is significant at a 95% confidence level. Below that, the gap may be due to chance. A winner is only named at the planned end of the test, or earlier once the orders needed are collected, but never before 2 weeks. A test planned for 1 week concludes at its end date. A test stopped before that only gives a trend.
• When the two groups don't have the expected share of visitors, a banner at the top of the page says so: a cache, a script or another app may be skewing the split. Don't trust the result yet, and use its Contact support button.

Previous tests stay listed under Previous tests, each with a verdict badge: Bar wins, Better without the bar, No difference or Inconclusive. Tests that ended before these badges existed don't have one. You can delete a test once it has ended.

After the test

When the test ends, every visitor sees the bar again: there is nothing to switch back. If the bar earned more, keep it as it is. If your store did better without it, the result page points you to your tiers: a threshold set too high can put some customers off. If the test was inconclusive, Relaunch with these settings opens the launch form with the same split, difference to detect and choice of what visitors without the bar lose.