Toward an Empirical Practice in Offline Fully Automatic Design of Robot Swarms
针对机器人群体优化设计中缺乏系统实证分析的问题，提出一个实验协议，包括任务生成器和最小化性能估计方差的采样策略，并通过比较两种离线全自动设计方法进行说明。
Due to the lack of systematic empirical analyses and comparisons of ideas and methods, a clearly established state of the art is still missing in the optimization-based design of robot swarms. In this article, we propose an experimental protocol for the comparison of fully automatic design methods. This protocol is characterized by two notable elements: 1) a way to define benchmarks for the evaluation and comparison of design methods and 2) a sampling strategy that minimizes the variance when estimating their expected performance. To define generally applicable benchmarks, we introduce the notion of mission generator: a tool to generate missions that mimic those a design method will eventually have to solve. To minimize the variance of the performance estimation, we show that, under some common assumptions, one should adopt the sampling strategy that maximizes the number of missions considered—a formal proof is provided as the supplementary material. We illustrate the experimental protocol by comparing the performance of two offline fully automatic design methods that were presented in previous publications.