Automating the quality prediction of survey questions: predicting measurement quality directly from natural text
研究评估了能否直接从调查问题的自然语言文本预测测量质量，发现微调语言模型（xlm-RoBERTa）的效果与基于人工编码的随机森林模型相当，可简化现有SQP工具的使用流程。
Abstract The survey quality predictor (SQP) is a web-based tool designed to predict the measurement quality of survey questions based on up to 72 manually coded formal and linguistic characteristics (e.g. domain, response scale properties, linguistic complexity). Users must input these features following a detailed coding manual, after which a trained random forest (RF) model predicts measurement quality. Here, we evaluate whether measurement quality can instead be predicted directly from the natural language text of a survey question, eliminating the need for manual coding. We find a fine-tuned language model can predict survey item quality based solely on the question and answer options, achieving performance comparable to the RF model currently implemented in SQP, which is trained on manually coded features. Specifically, we fine-tuned xlm-RoBERTa, a multilingual transformer-based model trained on multiple text corpora in over 100 languages, using the SQP dataset. Our findings suggest the current SQP web interface (https://sqp.gesis.org), which requires users to manually code the 72 features, can be simplified. A redesigned, more user-friendly web interface could allow users to only enter the survey question and answer options, with the model automatically predicting measurement quality. This would enhance accessibility and lower the barrier for researchers and practitioners using SQP.