A novel agreement statistic using data on uncertainty in ratings
针对现有一致性统计量校正机遇一致性时假设可能不适用的问题，提出一种利用评分者不确定性数据估计经验机遇一致性的新统计量，并推导其标准误估计量，模拟显示该统计量在多数情况下无偏。
Many existing methods for estimating agreement correct for chance agreement by adjusting the observed proportion agreement by the probability of chance agreement based on different assumptions. These assumptions may not always be appropriate, as demonstrated by pathologists' ratings of kidney biopsy descriptors. We propose a novel agreement statistic that accounts for the empirical probability of chance agreement, estimated by collecting additional data on rater uncertainty for each rating. A standard error estimator for the proposed statistic is derived. Simulation studies show that in most cases, our proposed statistic is unbiased in estimating the probability of agreement after removing chance agreement.