Large Language Models and Creative Content Design: a case study of email marketing at Wine Access
通过三轮随机对照试验，比较人工、大型语言模型及人机混合三种方式生成的电子邮件营销内容对小型在线企业利润的影响，发现AI方案在净年利润上优于纯人工方案。
Abstract A sequence of three randomized controlled trials (RCTs) is conducted to support a small online business’ decision of whether and how to implement AI in the creation of email marketing content. Recent developments in frequentist statistical decision theory are used to accommodate small samples available for testing in the small-business setting. The RCTs comprise three test policy cells with email content created by (i) salaried writers (“human”), (ii) a large-language model (“LLM”), and (iii) a “hybrid” combination of a human editing the content created by the LLM, respectively. When a “no email” control policy is included, all three test cells approximately double the gross profits from orders relative to the control cell. The RCTs vary whether the hybrid cell is edited by a salaried writer or the marketing team. The LLM cells vary whether the AI is pre-trained using historic emails or uses a prompt-based generative pre-trained transformer (“GPT”). Decision theory always selects one of the AI cells over the standard human policy on the basis of total annual profit net of related labor and software overhead.