When AI Becomes Its Own Biggest Fan: Self-Preference Bias in AI-Assisted Peer Review
研究了GPT-4、GPT-3和LLaMA在辅助同行评审时是否偏爱自己生成的文本，发现前两者存在显著自我偏好，而人类评审更接近AI交叉评审结果。
This study examined whether large language models exhibit self-preference bias, a tendency to favor texts generated by themselves over those produced by other models, in AI-assisted peer review. Using a controlled review simulation, GPT-4, GPT-3, and LLaMA were used to generate and evaluate 60 academic manuscripts under a common rubric covering clarity, relevance, originality, and methodological appropriateness. This design enabled direct comparison between self-reviews (a model evaluates its own generated manuscript) and cross-reviews (a model evaluates manuscripts generated by other models) under identical evaluation conditions. In addition, BERT was used to compute embedding-based semantic similarity as a robustness measure. The results indicated that GPT-4 and GPT-3 exhibited strong self-preference bias, assigning significantly higher scores to their own outputs than to manuscripts generated by other models. Human review scores were more closely aligned with AI cross-reviews than AI self-reviews, whereas LLaMA showed weaker and less consistent self-preference patterns, corresponding to its higher rejection rate during quality screening. These findings remained robust across prompt-sensitivity, resampling, alternative aggregation, subsample, and model-specification checks. Taken together, the findings point to a structural source of distortion in AI-assisted evaluation, suggesting that bias may arise not only from training data but also from how generation and evaluation roles are configured within a workflow. The study contributes to research on AI-enabled evaluation systems by extending current discussions of algorithmic bias toward workflow design, and it offers practical guidance on cross-model evaluation, calibration, and human-in-the-loop safeguards in high-stakes evaluative settings.