Three Things to Keep In Mind When Running eCommerce AB Tests

Ben Staples
4 min readMar 24, 2021

Originally published on www.Ben-Staples.com

Scientific based AB testing is a skillset critical to any eCommerce Product Manager. There is a lot that can go wrong with AB tests. A few weeks back I published an article about the novelty effect; a risk you can run into when not looking at either the length of time you’re running your test for, or the distribution of customer cohorts.

Besides the novelty effect, here are three things to keep in mind to avoid common pitfalls when running AB tests:

1. Isolate variables of change

The waterfall methodology incentivizes big bang projects where an end to end project plan is created, timelines are tacked on, and a whole wheelbarrow-full of change is dumped on the customer all at once. Changing 5–10 different variables all at the same time doesn’t work well with AB tests.

When you change so many variables at once, how will you be able to determine what drove what change to customer behavior or business metrics? You can’t. There is no way.

Agile practices push for incremental changes. By adjusting one variable at a time and running very quick iterations, you can see the resulting customer impact and directly correlate it to the user experience changes you’re…

--

--

Ben Staples

Ben Staples has over 7 years of product management and PMM eCommerce experience from at Nordstrom, Trunk Club, and Vistaprint . Currently based in Chicago IL