潜在ノイズ表現ブリッジによる拡散ポリシーの効率的ファインチューニング
Efficient Diffusion Policy Fine Tuning with Latent Noise Representation Bridging
拡散・フローポリシーの強化学習による適応において、ノイズ空間と行動空間を共有表現で結ぶブリッジ方程式を導出し、ノイズエイリアシング問題を解消するLARBRIDアルゴリズムを提案した。歩行・操作タスクで2〜3倍の性能向上を達成。
詳しい要約
1. どんなもの？
2. 先行研究と比べてどこがすごい？
3. 技術・手法の肝は？
4. どうやって有効だと検証した？
5. 議論はある？
6. 次に読むべき論文は？
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Yuanchu Liang, Yue Yang, An-Chi He, Hanna Kurniawati
分類: cs.RO, cs.AI
原文アブストラクト
Pre-trained diffusion and flow policies have emerged as powerful backbones for robotics, yet adapting them to novel data via reinforcement learning (RL) remains computationally expensive, sample inefficient and lacks theoretical guarantees. Prior approaches that fine-tune the initial noise bypass costly backpropagation through time but struggle with the "noise aliasing" problem: there does not exista natural link between the noise space value and the action space value. This paper resolves these challenges by introducing the Latent Noise Linear Markov Decision Process (LNL-MDP). We derive the Bridge Equation for LNL-MDPs which eliminates the noise aliasing problem by aligning values with a shared representation between the noise and the action space. Furthermore, our LNL-MDP fine tuning framework falls under the FineTuneRL theory with near-optimal guarantee. Leveraging these insights, we propose Latent Representation Bridging for Diffusion (LARBRID), an algorithm that efficiently learns the representation to optimize the initial distribution that steers the pre-trained model to act optimally in the environment. It yields up to 2-3$\times$ performance increases on challenging control problems across a diverse suite of robotic locomotion and manipulation tasks. Code can be found at: github.com/Maincode-Research/LARBRID.git.
関連論文
- DIA: 拡散ポリシー最適化のためのデノイジング中間アドバンテージ拡散ポリシー/強化学習