Comprehensive Evaluation of Explanation Types in a Spaceflight-Relevant Human–Autonomy Teaming Task
研究了在航天模拟器中，不同解释类型（全局、对比、演绎）对人机协作任务的表现、工作负荷、信任、情境意识和偏好的影响，发现对比加全局解释组合效果最佳。
Objective This study evaluates how explanation type in an explainable AI (XAI) human–autonomy teaming (HAT) task affects performance, workload, trust, situation awareness (SA), and preference in a dynamic, spaceflight-relevant simulator. Second, we introduce a holistic evaluation method for comparing XAI systems across multiple outcomes. Background XAI aims to improve understanding, calibrate trust, and enhance performance of an HAT, but the impact of explanation type in realistic, high-taskload HAT settings remains underexplored. Method Participants ( <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mrow> <mml:mi>N</mml:mi> <mml:mo>=</mml:mo> <mml:mn>31</mml:mn> </mml:mrow> </mml:math> ) completed 18 trials in a dual-task simulator requiring manual rover driving while supervising an autonomous exploration agent. Participants received various combinations of global, contrastive, and deductive explanations for AI-generated routes, with incentives tied to performance. Results Explanation type significantly affected manual performance ( <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mrow> <mml:mi>p</mml:mi> <mml:mo>=</mml:mo> <mml:mn>0.0003</mml:mn> </mml:mrow> </mml:math> ), autonomy performance ( <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mrow> <mml:mi>p</mml:mi> <mml:mo><</mml:mo> <mml:mn>0.0001</mml:mn> </mml:mrow> </mml:math> ), team performance ( <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mrow> <mml:mi>p</mml:mi> <mml:mo><</mml:mo> <mml:mn>0.0001</mml:mn> </mml:mrow> </mml:math> ), workload ( <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mrow> <mml:mi>p</mml:mi> <mml:mo><</mml:mo> <mml:mn>0.0001</mml:mn> </mml:mrow> </mml:math> ), trust ( <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mrow> <mml:mi>p</mml:mi> <mml:mo><</mml:mo> <mml:mn>0.0001</mml:mn> </mml:mrow> </mml:math> ), and preference ( <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mrow> <mml:mi>p</mml:mi> <mml:mo>=</mml:mo> <mml:mn>0.001</mml:mn> </mml:mrow> </mml:math> ), but not SA ( <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mrow> <mml:mi>p</mml:mi> <mml:mo>=</mml:mo> <mml:mn>0.41</mml:mn> </mml:mrow> </mml:math> ). Participants preferred global and contrastive explanations, performing better with their preferred explanation ( <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mrow> <mml:mi>p</mml:mi> <mml:mo>=</mml:mo> <mml:mn>0.049</mml:mn> </mml:mrow> </mml:math> ). Conclusion Explanation type influences performance and perception in demanding HAT contexts. A standardized, multi-metric evaluation framework is essential for understanding tradeoffs in XAI design. Application In HAT tasks like space exploration where users must quickly make decisions with an AI teammate, designers must consider the explanation method for XAI explanations. Our human-centered evaluation found a contrastive + global explanation combination was the best in our HAT task across a range of performance and preference metrics.