Deep Learning From Crowds on a Healthy Data Diet
针对众包数据噪声大、冗余多的问题，提出CrowdSketch样本选择方法，通过局部对比损失和预测分歧度筛选高质量数据，并基于误差L2范数去除冗余，在LabelMe和CIFAR-10H上分别提升准确率1.27%和1.52%。
Learning from crowds aims to train a robust and generalizable model with a noisy crowdsourced dataset from multiple annotators. Due to its simplicity and practicality, the target model and label correction mechanism are co-trained in a parameter-coupled manner. However, this end-to-end training suffers from the inherent flaws in data. The crowdsourced dataset is naturally noisy and redundant: 1) some annotations may violate the annotator’s transition matrix and mislead the training procedure and 2) the size of the annotation set is much larger than the instance set, and some instances may be backpropagated multiple times without performance gain. To address these issues, we propose a sample selection method called CrowdSketch for deep learning from crowds. Specifically, to mitigate the annotation noise, we find possibly high-quality data with small local contrastive loss (clean) and high divergence loss between prediction probabilities (important) on a two-branch network. After that, to alleviate redundant data, an importance score is developed based on <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</i><sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub>-norm of errors. The originality of this work stems from the designed selection criteria and specified two-branch architecture for crowdsourcing. Besides, by removing the noisy data, the risk of nonoptimum in dataset pruning is reduced. Extensive experiments are conducted on real-world crowdsourcing datasets. The experimental results, which show average accuracy improvements of 1.27% on LabelMe and 1.52% on CIFAR-10H, demonstrate the effectiveness of CrowdSketch.