マニピュレーション2026/9/14arXiv:2609.14878

MPCを足場にした実世界強化学習による巧みな操作

Real-World Reinforcement Learning with MPC Scaffolding for Dexterous Manipulation

サンプリングベースMPCを足場として実機の巧みなハンド操作を強化学習し、短時間で高成功率と高速回転を実現した。

詳しい要約

1. どんなもの？

- 実世界RLでdexterous manipulationを学習する枠組み - sampling-based MPCをscaffoldingとして利用 - 人間のdemonstrationやcorrective actionを不要とする - Allegro hand (16-DoF)でin-hand rotationを対象 - 7分のonline RLでpolicy-only評価100%成功(5/5) - 20分後にはMPCの5倍以上の回転速度を達成

2. 先行研究と比べてどこがすごい？

- 従来の実世界RLは初期探索が非効率で失敗コストが高い - 人間のdemonstrationやcorrective actionに依存する手法と異なる - MPCをscaffoldingとして構造化されたprior experienceを提供 - 人間介入なしで低interventionな実世界dexterous RLを実現 - 20 MPC trajectories (12分)の初期化で7分後に100%成功 - 20分でMPCの5倍以上の回転速度を達成

3. 技術・手法の肝は？

- sampling-based MPCをscaffoldingとして使用 - 少数のMPC trajectoriesでoffline replay bufferを初期化 - actorとcriticをpretrain - online学習中はMPCが断続的にdata collectionをガイド - off-policy Soft Actor-Criticでprior MPC経験と新規物理interactionから学習 - 制御は徐々に学習済みpolicyへ移行 - ablationでMPC pretraining、MPC経験保持、online MPC guidanceの相補的効果を確認

4. どうやって有効だと検証した？

- Allegro hand (16-DoF)でのcontinuous in-hand rotation - 20 MPC trajectories (hardwareで12分収集)で初期化 - 7分のonline RL後、policy-only評価で100%成功(5/5 trials) - online training中の平均object dropsは約3回 - 20分のonline learning後、MPCの5倍以上の回転速度 - 110分以上、1000回連続回転をdropなしで達成 - 異なるobject geometriesへの迅速な適応とgoal-conditioned reorientationを実証

5. 議論はある？

- ablationによりMPC-based pretraining、retained MPC experience、online MPC guidanceの相補的利点を確認 - 異なるobject geometriesへの迅速な適応を実証 - goal-conditioned reorientationの成功を実証 - 効率的で低interventionな実世界dexterous RLを可能にする - 具体的な限界や失敗事例の議論は要旨からは不明

6. 次に読むべき論文は？

- sampling-based model predictive control (MPC) - Soft Actor-Critic (SAC) - off-policy reinforcement learning - dexterous manipulation - in-hand rotation - Allegro hand - goal-conditioned reorientation - 要旨で参照/比較されている個別の先行研究は明示されていない

※ AIが要旨から生成した要約です。正確性は原文をご確認ください。

著者: Emek Barış Küçüktabak, Karankumar Patel, Zhaodong Yang, Jinda Cui, Kazuhiro Sasabuchi, Jun Takamatsu

分類: cs.RO

原文アブストラクト

Real-world reinforcement learning (RL) offers a promising route to dexterous manipulation policies that can adapt directly from physical interaction, but learning is hindered by inefficient early exploration and costly failures. We propose a framework that uses sampling-based model predictive control (MPC) as scaffolding for real-world dexterous RL, providing structured prior experience and task-directed guidance during learning without human demonstrations or corrective actions. A small set of MPC trajectories is first used to populate an offline replay buffer and to pretrain the actor and critic. During online learning, MPC intermittently guides data collection while an off-policy Soft Actor-Critic learner trains from both prior MPC experience and newly collected physical interaction, with control gradually transitioning to the learned policy. On continuous in-hand rotation with a 16-DoF Allegro hand, the method reaches 100\% success in policy-only evaluation (5/5 trials) after 7 minutes of online RL, following initialization with 20 MPC trajectories collected on hardware in 12 minutes. Online training incurs about three object drops on average. After 20 minutes of online learning, the policy achieves more than five times the rotation speed of the MPC controller. It completes 1000 consecutive rotations over more than 110 minutes without a drop. Ablations show complementary benefits from MPC-based pretraining, retained MPC experience, and online MPC guidance. We further demonstrate rapid adaptation to different object geometries and successful goal-conditioned reorientation, showing that the framework enables efficient, low-intervention, real-world dexterous RL.

arXivで原文を読む →arXiv概要ページ