The use of LLMs to annotate data in management research: Foundational guidelines and warnings
为管理研究者使用大语言模型进行数据标注提供基础框架和指南，通过分析众筹项目中的可持续性声明，展示LLM可媲美传统方法且成本更低，但提示设计的变化会显著影响结果，并给出敏感性分析和文档化建议。
Abstract Research Summary The emergence of large language models (LLMs) offers new opportunities for AI integration in research, particularly for data annotation and text classification. However, researchers lack guidance on implementation best practices, as the benefits and risks of these tools remain poorly understood. We develop a foundational framework for effective LLM implementation in management research, providing structured guidance on key decisions throughout the research process. We illustrate this framework through an empirical application: classifying sustainability claims in crowdfunding projects. While LLMs can match or exceed traditional methods' performance at lower cost, we find that variations in prompt design can significantly affect results and downstream analyses. We develop procedures for sensitivity analysis and provide detailed documentation to help researchers implement robustness while maintaining methodological integrity. Managerial Summary Large language models (LLMs) offer powerful new tools for business research, especially for analyzing and categorizing text data. However, managers and researchers lack clear guidance on how to use these tools effectively and reliably. This study creates a practical framework for implementing LLMs in management research, covering key decisions from model selection to result validation. Using a real‐world example of analyzing sustainability claims in crowdfunding campaigns, we demonstrate that LLMs can match traditional methods while being faster and cheaper. However, small changes in how you instruct the AI can significantly alter results and business conclusions. We provide systematic procedures for testing result reliability and offer practical tools to help managers implement AI‐powered analysis while maintaining rigorous standards and avoiding misleading findings.