接触の多い微分可能シミュレーションにおける適応的ホライズンアクター・クリティック
Adaptive Horizon Actor-Critic for Policy Learning in Contact-Rich Differentiable Simulation
剛体接触を含む微分可能シミュレーションで、モデルベースのホライズンを適応的に調整して勾配誤差を減らす強化学習アルゴリズムを提案し、歩行タスクで既存手法より40%高い報酬を達成した。
著者: Ignat Georgiev, Krishnan Srinivasan, Jie Xu, Eric Heiden, Animesh Garg
分類: cs.LG, cs.AI
原文アブストラクト
Model-Free Reinforcement Learning (MFRL), leveraging the policy gradient theorem, has demonstrated considerable success in continuous control tasks. However, these approaches are plagued by high gradient variance due to zeroth-order gradient estimation, resulting in suboptimal policies. Conversely, First-Order Model-Based Reinforcement Learning (FO-MBRL) methods employing differentiable simulation provide gradients with reduced variance but are susceptible to sampling error in scenarios involving stiff dynamics, such as physical contact. This paper investigates the source of this error and introduces Adaptive Horizon Actor-Critic (AHAC), an FO-MBRL algorithm that reduces gradient error by adapting the model-based horizon to avoid stiff dynamics. Empirical findings reveal that AHAC outperforms MFRL baselines, attaining 40% more reward across a set of locomotion tasks and efficiently scaling to high-dimensional control environments with improved wall-clock-time efficiency.