Specification Searching and Significance Inflation Across Time, Methods and Disciplines
利用20种发展项目的效果评估数据,发现随机对照试验的结果比其他方法更少出现显著性膨胀,且随时间推移其膨胀程度降低,而准实验研究则未改善。
Abstract This paper examines how significance inflation has varied across time, methods and disciplines. Leveraging a unique data set of impact evaluations on 20 kinds of development programmes, I find that results from randomized controlled trials exhibit less significance inflation than results from studies using other methods. Further, randomized controlled trials have exhibited less significance inflation over time, but quasi‐experimental studies have not. There is no robust difference between results from researchers affiliated with economics departments and those from researchers affiliated with other predominantly health‐related departments. Overall, the biases found appear much smaller than those previously observed in other social sciences.