On Machine Learning models explainability in the banking sector: the case of SHAP
研究了SHAP在银行信用风险模型中的可解释性，发现其解释结果受样本分布影响，可能随时间或银行不同而变化，对客户和监管构成问题。
Machine Learning models explainability has recently become a very popular topic in the banking sector. We apply Shapley values and the Python library SHAP to a real credit risk database and show that the two available SHAP types (interventional and path-dependent) provide very similar results even under correlated features. The main drawback of SHAP is that it is not portfolio invariant, this is, the explanation for the prediction provided for each observation depends on the portfolio distribution of the features. This can be a serious problem for customers and banking regulators, who expect that explanations will stay stable as long as the clients characteristics do not change. We conduct several tests and show that the SHAP explanation of an observation may considerably change depending on the rest of the portfolio distribution. As a consequence, the explanation of a client may vary over time even if her characteristics do not change and banks using the same model (for ex. commercial models) may provide different explanations to the same client.