強化学習における計算効率の良い安全な探索
Computationally efficient safe exploration in reinforcement learning
ガウス過程より軽量なNadaraya-Watson推定量を用いて、制約付きMDPを安全に探索・最適化するアルゴリズムCoLSafe-MDPを提案し、グリッド環境と火星地形データで評価した。
著者: Shreeram Murali, Shankar A. Deka, Dominik Baumann
分類: cs.LG, cs.RO, eess.SY
原文アブストラクト
Reinforcement learning in real-life applications requires safety guarantees during exploration. Typical reinforcement learning algorithms do not provide such guarantees, and many modifications that do rely on Gaussian processes (GPs), which have a large computational cost. We propose a computationally lightweight algorithm based on the Nadaraya-Watson estimator that safely explores and optimizes constrained Markov decision processes (MDPs). Our algorithm, \textsc{CoLSafe-MDP}, uses an estimator that scales in constant-time with bounds on the estimates, a significant improvement from its GP-based counterparts that scale cubically with the number of data points. We then evaluate its performance in a grid-based environment and on observational Martian terrain data.