過大評価バイアス制御の自動化による強化学習の効率化
Automating Control of Overestimation Bias for Reinforcement Learning
強化学習における過大評価バイアス制御のハイパーパラメータをデータ駆動で自動選択する手法を提案し、3つのアルゴリズムで有効性を示した。
著者: Arsenii Kuznetsov, Alexander Grishin, Artem Tsypin, Arsenii Ashukha, Artur Kadurin, Dmitry Vetrov
分類: cs.LG, cs.AI, cs.RO, stat.ML
原文アブストラクト
Overestimation bias control techniques are used by the majority of high-performing off-policy reinforcement learning algorithms. However, most of these techniques rely on pre-defined bias correction policies that are either not flexible enough or require environment-specific tuning of hyperparameters. In this work, we present a general data-driven approach for the automatic selection of bias control hyperparameters. We demonstrate its effectiveness on three algorithms: Truncated Quantile Critics, Weighted Delayed DDPG, and Maxmin Q-learning. The proposed technique eliminates the need for an extensive hyperparameter search. We show that it leads to a significant reduction of the actual number of interactions while preserving the performance.