割引なしマルコフ決定過程のための最小相対エントロピー制御器
A Minimum Relative Entropy Controller for Undiscounted Markov Decision Processes
未知のダイナミクスを持つ割引なしMDPに対し、相対エントロピー最小化に基づくベイズ制御則(BCR-MDP)を導出し、方策空間上のノンパラメトリック共役事前分布とギブスサンプラーを提案した。探索と活用のバランスにより準最適な極限周期を回避できることを示した。
著者: Pedro A. Ortega, Daniel A. Braun
分類: cs.AI, cs.LG, cs.RO
原文アブストラクト
Adaptive control problems are notoriously difficult to solve even in the presence of plant-specific controllers. One way to by-pass the intractable computation of the optimal policy is to restate the adaptive control as the minimization of the relative entropy of a controller that ignores the true plant dynamics from an informed controller. The solution is given by the Bayesian control rule-a set of equations characterizing a stochastic adaptive controller for the class of possible plant dynamics. Here, the Bayesian control rule is applied to derive BCR-MDP, a controller to solve undiscounted Markov decision processes with finite state and action spaces and unknown dynamics. In particular, we derive a non-parametric conjugate prior distribution over the policy space that encapsulates the agent's whole relevant history and we present a Gibbs sampler to draw random policies from this distribution. Preliminary results show that BCR-MDP successfully avoids sub-optimal limit cycles due to its built-in mechanism to balance exploration versus exploitation.
