Distilling Ensemble Surrogates for Federated Data-Driven Many-Task Optimization
针对联邦学习中客户端数据分布不同导致优化性能下降的问题，提出通过聚合局部模型构建集成代理，再用知识蒸馏得到全局代理，降低通信成本并提升模型泛化能力，实验验证了方法的有效性。
Black-box optimization problems are commonly seen in the real world, ranging from experimental design to hyperparameter tuning of machine learning models. In numerous scenarios, addressing a collection of similar data-driven black-box optimization tasks distributed on multiple clients not only raises privacy concerns but also suffers from non-independent and identically distributed (non-IID) data, seriously deteriorating the optimization performance. To address the above challenges, this article focuses on handling non-IID data in federated data-driven many-task optimization. To construct a high-quality global surrogate by robustly aggregating the local models, the server first fits a Gaussian distribution for each model parameter upon receiving local parameters, from which an ensemble model can be sampled. To reduce the communication cost and provide a generalized global model, a student surrogate model is derived by means of knowledge distillation from the ensemble. In addition, each client is allowed to retain both local and global models, so that the mean and variance of the predictions can be used to guide the selection of new samples. Experimental results demonstrate the reliability and efficacy of our proposed method on both benchmark problems and a real machine learning problem in the presence of non-IID data.