Reinforcement Learning with Limited Reinforcement: Using Bayes Risk for Active Learning in POMDPs

Partially Observable Markov Decision Processes (POMDPs) have succeeded in planning domains because they optimally trade between actions that increase an agent's knowledge and actions that increase an agent's reward. Unfortunately, most POMDPs are defined with a large number of parameters which are difficult to specify only from domain knowledge. In this paper, we treat the POMDP model parameters as additional hidden state in a "model-uncertainty" POMDP and develop an approximate algorithm for planning in the this larger POMDP. The approximation, coupled with model-directed queries, allows the planner to actively learn good policies. We demonstrate our approach on several standard POMDP problems.