Get the latest tech news
Five ways to reduce variance in A/B testing
I use toy Monte Carlo simulations to demonstrate 5 ways to reduce variance in A/B testing: increase sample size, move towards a more even split, reduce variance in the metric definition, stratification and CUPED.
increase sample size move towards an even split reduce variance in the metric definition stratification CUPED It's worth noting that I used Monte Carlo simulations over many experiments (typically 1,000) to show the decrease in variance, by visualizing the spread of the lift histogram. The rewards of lower variance in A/B testing are reaped in the long term, by having more accurate measurements, making better decisions and releasing better products more quickly.
Or read this on Hacker News
