Comparing and Improving Active Learning Uncertainty Measures for Transformer Models by Discarding Outliers
比较了八种主动学习不确定性度量方法在七个数据集上的表现，发现softmax函数提供误导性概率，多数方法倾向于识别难以学习的异常值而非降低模型不确定性的样本，并提出一种启发式方法Uncertainty-Clipping来系统排除异常值以提升性能。
Abstract Despite achieving state-of-the-art results in nearly all Natural Language Processing applications, fine-tuning Transformer-encoder based language models still requires a significant amount of labeled data to achieve satisfying work. A well known technique to reduce the amount of human effort in acquiring a labeled dataset is Active Learning (AL): an iterative process in which only the minimal amount of samples is labeled. AL strategies require access to a quantified confidence measure of the model predictions. A common choice is the softmax activation function for the final Neural Network layer. In this paper, we compare eight alternatives on seven datasets and show that the softmax function provides misleading probabilities. Our finding is that most of the methods primarily identify hard-to-learn-from samples (commonly called outliers), resulting in worse than random performance, instead of samples, which actually reduce the uncertainty of the learned language model. As a solution, this paper proposes Uncertainty-Clipping, a heuristic to systematically exclude samples, which results in improvements for most methods compared to the softmax function.