On Average Deviation Indices for Estimating Interrater Agreement
通过两项针对零售公司销售员工的调查，检验了基于均值和中位数平均偏差的两个评分者间一致性指标，发现它们与其他指标高度相关，支持其实际应用。
In this investigation, the authors report the results of two studies designed to investigate the efficacy of two proposed indices of interrater agreement based on average deviations from the mean and from the median (ADM and ADMd , respectively). Using survey response data collected from 6,549 sales employees in 119 stores of a national retail company, Study 1 compared the results of six interrater agreement indices across four types of Likert-type response scales (i.e., 5-, 6-, 7-, and 11-point scales). The results indicated that the AD indices were highly correlated with an index of proportional agreement and with within-group interrater agreement indices. Study 2, based on survey data collected from 4,158 sales employees in 109 other stores of this company, constructively replicated Study 1 and examined the consistency of interrater agreement decisions across six indices with respect to a priori decision rules. Study 2 results also supported the use of AD indices. Practical issues concerning the use of AD indices for estimating interrater agreement and future research directions are discussed.