潜在想像によるパラメータ化行動空間の学習
Latent Imagination for Parameterized Action Spaces
世界モデルの潜在空間内で離散行動と連続パラメータを同時に選ぶハイブリッド方策を学習し、1回の順伝播で高速に行動決定するモデルベース手法HyLIを提案。7つのベンチマークでモデルフリー最強手法と同等以上の性能をより少ない環境ステップで達成し、プランナーより67倍高速に判断できる。
著者: Atahan Cilan, İhsan Mert Muhacıroğlu, Ümit Can Bekar
分類: cs.LG, cs.AI
原文アブストラクト
Parameterized-action MDPs (PAMDPs) ask an agent to choose a discrete action together with continuous parameters whose meaning depends on that choice. Model-free PAMDP methods act fast but need many interactions; model-based methods are sample-efficient but plan at every decision, which makes them slow to act. We introduce HyLI (Hybrid Latent Imagination), a model-based agent that learns a hybrid policy entirely in the latent imagination of a world model and acts with a single forward pass. To address learning problems that the PAMDP structure creates, HyLI adds counterfactual mode evaluation, which gives every discrete action a learning signal at every imagined state, and success-window replay, which keeps rare successes in training. On seven standard PAMDP benchmarks, HyLI is within one published standard deviation of, or above, the published score of the strongest model-free baseline in our comparison on all 7 tasks, reaches 90% of that score with 2.4x fewer environment steps than that baseline (5.0x when its reward-free pre-training is counted), and decides 67x faster than the model-based planner DLPA at its published planner settings. It is the only method in our comparison that is both more sample-efficient than that baseline and fast enough for real-time control.