AI-Augmented Content Validation in Behavioral Research: Development and Evaluation of the RATER System
开发了一个免费网页系统RATER，利用AI模型评估测量工具的内容效度，帮助研究者快速判断量表条目是否准确对应目标构念，经六项研究验证其准确性和可靠性。
Content validation is an essential aspect of the scale development process that ensures that measurement instruments capture their intended constructs. However, researchers rarely undertake this core step in behavioral research because it requires costly data collection and specialized expertise. We present RATER (replicable approach to expert ratings), a free web-based system (www.contval.org) that can help the broader research community (scientists, reviewers, students) gain quick and reliable insights into the content validity of measurement instruments. Guided by psychometric measurement theory, RATER evaluates whether a scale’s items correspond to their intended construct, remain distinct from other constructs, and adequately represent all aspects of the construct’s content domain. The system employs two unique artificial intelligence models, RATERC and RATERD, which leverage psychometric scales from 2,443 journal articles spanning eight disciplines and two state-of-the-art large language model architectures (i.e., BERT and GPT). A set of six complementary studies confirms the RATER system’s accuracy, reliability, and usefulness. We find that RATER can augment the scale development and validation process, increasing the validity of findings in behavioral research.