Three Things to Keep In Mind When Running eCommerce AB Tests
Originally published on www.Ben-Staples.com
Scientific based AB testing is a skillset critical to any eCommerce Product Manager. There is a lot that can go wrong with AB tests. A few weeks back I published an article about the novelty effect; a risk you can run into when not looking at either the length of time you’re running your test for, or the distribution of customer cohorts.
Besides the novelty effect, here are three things to keep in mind to avoid common pitfalls when running AB tests:
1. Isolate variables of change
The waterfall methodology incentivizes big bang projects where an end to end project plan is created, timelines are tacked on, and a whole wheelbarrow-full of change is dumped on the customer all at once. Changing 5–10 different variables all at the same time doesn’t work well with AB tests.
When you change so many variables at once, how will you be able to determine what drove what change to customer behavior or business metrics? You can’t. There is no way.
Agile practices push for incremental changes. By adjusting one variable at a time and running very quick iterations, you can see the resulting customer impact and directly correlate it to the user experience changes you’re…
