METHODOLOGICAL AND PSYCHOMETRIC ISSUES IN SETTING CUTOFF SCORES USING THE ANGOFF METHOD
研究了安戈夫法在标准参照测验中设定切分分数时的心理测量学问题，发现专家判断能提高数据准确性和可靠性，并展示了评分者分析技术和校准方法以优化标准设定。
The present study examined several psychometric issues relevant to the use of a favored technique (the Angoff method) used to set standards in criterion referenced testing. The research was conducted within a setting which allowed (a) confident identification of expert and non‐expert judges, and (b) estimation of “true” scores for items judged so that accuracy of judgments in addition to reliability could be examined. Results suggested that expertise of judges does make a difference in producing more accurate and reliable data, underscoring the importance of using true subject matter experts (SMEs) in the judgment process. A rater analysis technique (rater‐total correlations) was illustrated, which might prove useful in improving the quality of data obtained using the Angoff method, particularly when there is some question regarding the internal consistency of ratings and expertise of some of the raters. Finally, a rater accuracy adjustment/calibration technique was examined and proved to be a potentially useful method to maximize accuracy of a standard derived using the Angoff method in settings where archival normative test data can be obtained. Other methods that could potentially be used to improve Angoff data were discussed.