Assessing Within-Group Agreement
研究了组织心理学中常用的组内一致性指标r WG，重点比较了随机组重抽样（RGR）方法与模拟方法的优劣，并数学证明了RGR方法下统计显著性依赖于组内相关系数和组大小。
The measure of within-group agreement most frequently encountered in organizational psychology is the r WG index. The r WG index is determined by comparing the observed group variance among raters with an expected random variance. The most critical issue in calculating the r WG is the choice of an appropriate random distribution that would be expected to follow from raters making their ratings at random. A data-driven approach that uses random-group resampling (RGR) procedures to determine the expected random variance has been proposed. In the present study, the application of the RGR procedure will be illustrated with reference to students' ratings of their mathematics instruction and critically compared with a recently proposed simulation-based approach. It will be shown mathematically that the probability of obtaining statistically significant within-group agreement when applying the RGR procedure strongly depends on the intraclass correlation as well as on the group sizes. Finally, implications for applying the RGR procedure to assess within-group agreement in multilevel data will be discussed.