Operation Optimization Decision-Making of Aluminum Electrolysis Process Using Offline Reinforcement Learning
针对铝电解过程数据包含多种行为策略和风险策略的问题，提出一种带有多类别策略约束的离线多目标强化学习方法，在满足工业安全要求的同时超越原有策略，并通过真实数据实验验证了有效性。
A common scenario in aluminum electrolysis process is that the collected dataset contains different behavioral policies and some risky policies, such industrial scenario brings new challenges for offline reinforcement learning to learn safety and feasible optimization policy. This article proposes an offline multiobjective reinforcement learning with multicategory policy constraint for operation optimization decision-making (OODM) of the aluminum electrolysis process. The learned optimization policy can surpass the behavior policy while also meet the strict safety requirements of industrial operations. To alleviate the distribution shift problem on multicategory behavioral policy, we present a multicategory policy constraint in the actor network that utilizes the mixture Gaussian variational autoencoder (GMVAE) to implement behavior cloning between the behavioral policy and the learned policy. Based on the actor-critic reinforcement learning architecture, we design two critic networks for multiobjective optimization. Except for the operational performance critic network, an additional safety critic network is introduced to guarantee that the learned policy satisfies the strict safety requirements of industrial operations. We also conduct extensive comparative experiments on the real-world aluminum electrolysis process. Experimental results demonstrate that the proposed method can achieve superior performance against the other offline reinforcement learning algorithms.