Finding “similar” universities using ChatGPT for institutional benchmarking: A large‐scale comparison of European universities
研究测试了ChatGPT能否为大学找到科研表现相似的同行机构，发现直接使用会复制排名偏见，但语义关联能捕捉定量方法难以衡量的相似维度。
Abstract The study objective was to evaluate the efficacy of ChatGPT in identifying “similar” institutions for benchmarking the research performance of a university. Benchmarking is deemed a promising approach to compare “similar with similar” as a better alternative to rankings (comparing “different” universities). Current approaches either focus on a limited number of “quantitative” dimensions or are too complex for most users. We conducted large‐scale testing by tasking ChatGPT with identifying the most similar European universities in terms of research performance, utilizing the European Tertiary Education Register data. We tested whether the peers suggested by ChatGPT were similar to the focal university on size, research intensity, and subject composition. Additionally, we evaluated whether providing more specific instructions improved the results. The findings offer a nuanced perspective on the potential and risks of using ChatGPT to identify peer institutions for benchmarking. On one hand, solely using ChatGPT would replicate the visibility biases associated with university rankings, thereby undermining the rationale for benchmarking. On the other hand, relying on semantic associations might capture dimensions of university similarity that are relevant and difficult to capture through quantitative methods. We finally reflected on the broader implications for scholars in higher education and science studies research.