Do “Evidence‐Based Policy” Clearinghouses Provide Good Advice for Local Policymakers?
本文构建贝叶斯模型，检验“基于证据的政策”规则对地方政策制定者的建议质量，发现除非跨地区效果差异很小，否则该规则的正确决策概率、阳性预测值和阴性预测值都低得不可接受。
ABSTRACT Policymakers are often urged to rely on “evidence‐based policy” (EBP)—adopting only interventions proven effective (i.e., positive and statistically significant in multisite impact evaluations). EBP clearinghouses chronicle and rate tests of social policy interventions. But EBP clearinghouse standards are based almost entirely on internal validity. They largely ignore whether research findings from multisite trials apply to individual localities, where much of social policy is formulated. We develop a Bayesian model of the probability that the EBP rule is sound advice to local policymakers. The model allows a direct test of the probability of a correct policy decision under the EBP rule, its positive predictive value (PPV), and its negative predictive value (NPV)—the probabilities that an intervention deemed effective by that rule will in fact be effective in a particular site (PPV), and that an intervention deemed ineffective will not be effective in a particular site (NPV), given the true impact of the intervention. These intuitive, easily calculated probabilities are major contributions of this paper. In our illustrative analysis of six multisite randomized trials, we find that under the EBP clearinghouse rule the probability of a correct policy decision, PPV, and NPV are all unacceptably low unless the cross‐site impact heterogeneity is quite low.