A/B Testing with Fat Tails
提出厚尾分布下的最优实验框架，发现当创新质量分布厚尾时，采用更多小样本实验的“精益”策略优于传统大样本实验，并结合微软Bing平台数据验证了改进实践可提升创新效率。
We propose a new framework for optimal experimentation, which we term the “A/B testing problem.” Our model departs from the existing literature by allowing for fat tails. Our key insight is that the optimal strategy depends on whether most gains accrue from typical innovations or from rare, unpredictable large successes. If the tails of the unobserved distribution of innovation quality are not too fat, the standard approach of using a few high-powered “big” experiments is optimal. However, if the distribution is very fat tailed, a “lean” strategy of trying more ideas, each with possibly smaller sample sizes, is preferred. Our theoretical results, along with an empirical analysis of Microsoft Bing’s EXP platform, suggest that simple changes to business practices could increase innovation productivity.