Reproduce to validate: A comprehensive study on the reproducibility of economics research
研究了《美国经济杂志：应用经济学》自2009年实施数据可用性政策后，对363篇论文中的274篇进行复制评估，发现仅44.7%的尝试复制能完全复现结果，表明数据政策虽保障材料可用但不足以确保可复制性。
Abstract Journals have pushed for transparency of research through data availability policies. Such data policies improve availability of data and code, but what is the impact on reproducibility? We present results from a large reproduction exercise for articles published in the American Economic Journal: Applied Economics, which has had a data availability policy since its inception in 2009. Out of 363 published articles, we assessed 274 articles. All articles provided some materials. We excluded 122 articles that required confidential or proprietary data or that required the replicator to otherwise obtain the data (44.5% of assessed articles). We attempted to reproduce 152 articles and were able to fully reproduce the results of 68 (44.7% of attempted reproductions). A further 66 (43.4% of attempted reproductions) were partially reproduced. Many articles required complex code changes even when at least partially reproduced. We collect bibliometric characteristics of authors, but find no evidence for author characteristics as determinants of reproducibility. There does not appear to be a citation bonus for reproducibility. The data availability policy of this journal was effective to ensure availability of materials, but is insufficient to ensure reproduction without additional work by replicators.