Why Meta-Analysis Doesn’t Tell Us What the Data Really Mean: Distinguishing between Moderator Effects and Moderator Processes
指出传统元分析中基于残差方差的调节效应推断可能因统计伪像与真实调节变量共变而产生虚假结果，并通过同伴提名效度研究实例说明，当样本量与内容非随机选择时，元分析结果无法解释调节过程。
Traditional approaches to detecting the presence of moderators in meta-analyses involve inferences drawn from the residual variance in criterion-related validities (a) after correcting for sampling error and statistical artifacts. James, Demaree, Mulaik, and Ladd (1992) argued that these residualized interpretations of meta-analytic results may be spurious when statistical artifacts covary with true moderators. We extend their model to suggest that situational moderators might also covary with sample size and content (i.e., nonrandom sample selection error), causing meta-analysis to be uninterpretable and a significant correlation between criterion-related validities and ni. We investigate this possibility on studies examining criterion-related validities ofpeer nominations originally reported by Kane and Lawler (1978). Application of residualized meta-analysis suggests the presence of moderator effects, but a significant correlation between ri and ni precludes interpretation of the moderator process behind these effects. More generally, we argue that the nature of true contingencies cannot be inferred from meta-analytic summaries of traditional criterion-related validity studies. Primary research with appropriate controls is the only means of identifying true moderator effects and processes on criterion-related validity.