Can social media provide early warning of retraction? Evidence from critical tweets identified by human annotation and large language models
分析了3815条提及撤稿文章的推文和3373条提及未撤稿文章的推文，发现8.3%的撤稿文章在撤稿前有至少一条批评性推文，而对照组仅1.5%，表明社交媒体有预警潜力，但大语言模型自动识别需谨慎。
Abstract Timely detection of problematic research is essential for safeguarding scientific integrity. To explore whether social media commentary can serve as an early indicator of potentially problematic articles, this study analyzed 3815 tweets referencing 604 retracted articles and 3373 tweets referencing 668 comparable non‐retracted articles. Tweets critical of the articles were identified through both human annotation and large language models (LLMs). Human annotation revealed that 8.3% of retracted articles were associated with at least one critical tweet prior to retraction, compared to only 1.5% of non‐retracted articles, highlighting the potential of tweets as early warning signals of retraction. However, critical tweets identified by LLMs (GPT‐4o mini, Gemini 2.0 Flash‐Lite, and Claude 3.5 Haiku) only partially aligned with human annotation, suggesting that fully automated monitoring of post‐publication discourse should be applied with caution. A human–AI collaborative approach may offer a more reliable and scalable alternative, with human expertise helping to filter out tweets critical of issues unrelated to the research integrity of the articles. Overall, this study provides insights into how social media signals, combined with generative AI technologies, may support efforts to strengthen research integrity.