Toward Universal Controller: Performance-Aware Self-Optimizing Reinforcement Learning for Discrete-Time Systems With Uncontrollable Factors
针对工业系统中不可控变量（如天气、摩擦）影响控制性能的问题，提出一种性能感知自优化通用控制器，通过历史轨迹感知不可控变量影响并设计奖励函数，结合域随机化训练策略消除其影响，提升系统稳定性。
The industrial system usually contains not only controllable variables (CVs) but also uncontrollable variables (unCVs), e.g., weather conditions and friction. These unCVs have a direct impact on system control performance. Despite the success of current deep reinforcement learning (DRL) control algorithms, most of them neglect the impact of unCVs, which can cause the deterioration of control performance and instability of the system. To perceive and eliminate the impact of unCVs, a performance-aware self-optimizing universal controller (PASOUC) is designed in this article. The PASOUC aims at integrating the representation of unCVs and controller design to perceive and eliminate the impact of unCVs under different conditions, which goes beyond most existing control methods. Technically, a historical trajectory-inspired control performance perceptron is developed to perceive the impact of unCVs on system control performance under different conditions. Subsequently, a new performance-aware reward is designed to integrate the representation of unCVs and controller design while training the DRL controller. In addition, the domain randomization (DR) training strategy is employed to learn a universal control policy, which can access the approximate optimal trajectory under nonideal conditions. In this way, the impact of unCVs can be eliminated. To handle the low efficiency of the DR training, the policy improvement-policy proximal optimization (PI-PPO) is proposed to enhance the convergence speed of the DR training by performing explicit policy improvement. Finally, illustrative examples are presented to demonstrate the superiority of the proposed method.