Expectation-maximization for Bayes-adaptive POMDPs
针对贝叶斯自适应POMDP，提出两种期望最大化方法：基于采样的EM算法适用于小规模问题，而约束变分贝叶斯EM算法能在大规模问题中生成高质量解，解决了标准VB-EM对某类策略的偏好问题。
Bayes-adaptive POMDPs (BAPOMDPs) are partially observable Markov decision problems in which uncertainty in the state-transition and observation-emission probabilities can be captured by a prior distribution over the model parameters. Existing approaches to solving BAPOMDPs rely on model and trajectory sampling to guide exploration and, because of the curse of dimensionality, do not scale well when the degree of model uncertainty is large. In this paper, we begin by presenting two expectation-maximization (EM) approaches to solving BAPOMPs via finite-state controller (FSC) optimization, which at their foundation are extensions of existing EM algorithms for BAMDPs to the more general BAPOMDP setting. The first is a sampling-based EM algorithm that optimizes over a finite number of models drawn from the BAPOMDP prior, and as such is only appropriate for smaller problems with limited model uncertainty; the second approach leverages variational Bayesian methods to ensure tractability without sampling, and is most appropriate for larger domains with greater model uncertainty. Our primary novel contribution is the derivation of the constrained VB-EM algorithm, which addresses an unfavourable preference that often arises towards a certain class of policies when applying the standard VB-EM algorithm. Through an empirical study we show that the sampling-based EM algorithm is competitive with more conventional sampling-based approaches in smaller domains, and that our novel constrained VB-EM algorithm can generate quality solutions in larger domains where sampling-based approaches are no longer viable.