Thanks for sharing Stephen. I am not familiar with the Bonferroni Correction. Do you have any resources that might be good to learn more?

I'm also wondering, if you launched a "Coming soon" style feature in an AB test and ran it until you got statistical significance at a high enough confidence interval, would that in a way combat the novelty effect?

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store