チャンク単位のモデルベース方策最適化
Chunk-wise Model-based Policy Optimization
世界モデルと方策をチャンク単位で協調予測し、1回の順伝播でHステップの仮想ロールアウトを生成して価値展開を直接最適化するモデルベースRL手法を提案。
著者: Weixi Song, Donglin Wang, Yong-Lu Li
分類: cs.RO, cs.LG
原文アブストラクト
Model-based reinforcement learning has shown superior sample efficiency by leveraging the learned latent dynamics either in the form of derivative-free planning/imagined rollout training or dynamics gradient-based policy updates. While these approaches achieve comparatively fast learning, they all suffer, to varying degrees, from issues of recurrent compounding errors, high inference-time latency, and instability in backward propagation through time (BPTT). In this work, we propose a model-based RL approach that uses the chunk world model as a differentiable training-time evaluator to optimize the chunk actor without recurrent model invocation. Through the collaborative chunk-wise prediction of the world model and policy, we can generate H-step hypothetical rollouts in a single forward pass and directly optimize value expansion, avoiding the costly and unstable BPTT. We evaluate our method on 19 challenging high-dimensional humanoid tasks from HumanoidBench and deploy our method to the real-world for quadruped bipedal walking. Project Website: https://song-wx.github.io/cmpo.github.io/