Commentary on “Pitfalls of amateur regression: The Dutch New Herring controversies”
本文回应Gao和Gill对作者2022年研究的批评，指出其评论基于2017年的博客而非正式论文，并引用独立评审结论支持自身方法，强调利益冲突下专家评审的不可靠性。
In Vollaard and Van Ours (2022), we provide evidence of a sizeable bias in an influential nonblind expert review in the food service industry. Almost all of the discussion in Gao and Gill (2023) refers not to our article in the Journal of Economic Behavior and Organization, however, but to two blogs in Dutch language about the same case written by one of us and published in 2017. At the time, Gill was paid to assess these blogs by the tabloid newspaper that published the expert review, as part of a complaint filed by that party. The discussion in Gao and Gill (2023) is couched in highly tendentious terms. They suggest that the two blogs represent “immature work,” that the model used is “simply wrong and irresponsible,” and that our results are based on “amateur regression,” to name just a few of their derogatory claims. They describe the data and the approach used in these terms, state that there are “big dangers” associated with the kind of work done, but do not show that their points of criticism would lead to a different conclusion, which was phrased in the second blog as: “These points do not constitute direct evidence of favoritism to fishmongers supplied by the supplier with the conflict of interest, but the test team has all the appearances to the contrary based on this investigation.” Surprisingly, Gao and Gill (2023) mention the complaint on page 3, but do not refer to the review by independent scientists in response to the complaint at the request of LOWI, an independent advisory body established by the Royal Netherlands Academy of Arts and Sciences, the Netherlands Organization for Scientific Research, and Dutch universities (LOWI, 2019). This provides an independent perspective on the very same work that Gao and Gill review in their article and should therefore be of interest to those who wish to learn from “the Dutch New Herring controversies.” According to the independent reviewers, “the approach and manner in which the research was conducted do not indicate a violation of any principle of good scientific research” (LOWI, 2019). Based on this assessment, the complaint was declared unfounded. Toward the end of their paper, Gao and Gill discuss our paper Vollaard and Van Ours (2022). In that paper, we do make causal claims, based on an explicitly stated identifying assumption. To illustrate our careful line of reasoning in our 2022 paper we repeat some of it. One of the lines of research we followed was to create samples that were homogeneously judged in terms of degree of ripening, cleaning quality, whether the herring was freshly prepared, and a judgment of the taste, smell, visual appearance, and texture of the fish. Within such a homogeneous sample, there should be no difference in the ratings related to the wholesalers. It turns out that there is a difference. The supplier with the conflict of interest received an average (provisional) rating that was 0.5 points higher (on a scale of 0–10). This may not seem like much, but the average for the homogeneous sample was 9.31. Moreover, the fishmongers supplied by the supplier with the conflict of interest are on average ranked in the top 10 of the review while the other fishmongers with the same underlying ratings do not appear in the top 10 at all. Gao and Gill's comment on our analysis is that “it would not surprise us if a few misidentified outlets and slightly modified “verbal judgements” could ruin their conclusion.” Well, to be fair, it would not surprise us if the findings for the homogeneous sample confirmed that the tasting panel was indeed biased. Expert reviews with a built-in conflict of interest are never a good idea. This is also why we have concerns about Gao and Gill (2023), where one of the authors was paid to assess our work. If anything, their paper says more about the authors than about our work, which was both independently conducted and independently reviewed. We invite interested readers to review our 2022 study for themselves. We have made our data and code publicly available so that anyone can review our work and make their own judgments (Van Ours & Vollaard, 2022).