AI in innovation research: an overview of transformers
本文介绍了Transformer模型的核心组件及其在专利文本分析中的应用，重点展示了如何利用该模型衡量技术新颖性，为创新研究者提供实用工具。
Patent text documents are a valuable and increasingly important data source for researchers in the field of innovation. Recent advances in natural language processing—particularly those centred on Transformers—have opened radically new opportunities for extracting information from patent text. Transformers, large language models (LLMs) built on deep learning architectures, rely on key underlying components such as attention mechanisms and word embeddings that enable a semantic understanding that is unparalleled compared to traditional approaches to text analysis. As such, Transformers represent a fundamental leap in how innovation researchers can extract meaning from patent documents. In this paper, we bridge a technical and applied perspective by unpacking the core components of Transformers, drawing on essential concepts from machine learning and linguistic theory. We then illustrate how Transformers can be leveraged in patent research, highlighting several potential applications, with a focus on measuring technological novelty. To ground our discussion, we present some exploratory analyses to demonstrate how Transformers can be used in practice.