Three Things to Keep In Mind When Running eCommerce AB Tests
--
Originally published on www.Ben-Staples.com
Scientific based AB testing is a skillset critical to any eCommerce Product Manager. There is a lot that can go wrong with AB tests. A few weeks back I published an article about the novelty effect; a risk you can run into when not looking at either the length of time you’re running your test for, or the distribution of customer cohorts.
Besides the novelty effect, here are three things to keep in mind to avoid common pitfalls when running AB tests:
1. Isolate variables of change
The waterfall methodology incentivizes big bang projects where an end to end project plan is created, timelines are tacked on, and a whole wheelbarrow-full of change is dumped on the customer all at once. Changing 5–10 different variables all at the same time doesn’t work well with AB tests.
When you change so many variables at once, how will you be able to determine what drove what change to customer behavior or business metrics? You can’t. There is no way.
Agile practices push for incremental changes. By adjusting one variable at a time and running very quick iterations, you can see the resulting customer impact and directly correlate it to the user experience changes you’re making through statistically significant AB tests.
2. Keep your backlog flexible
A trap many Product Managers fall into often driven by organizational pressures, is creating massive backlogs with year long time horizons. If you plan everything out well in advance for the team, you won’t have the flexibility needed to react to test results.
By having a flexible backlog that can change, and appropriately setting the expectation that it will change continuously with stakeholders, you can set your team up for success.
The whole benefit of AB testing is getting early and often reads on how the end user will react to a new feature or change. Unless you can clearly capitalize on those learnings and adjust…
