フローベースVLAポリシーのための成果均衡PPOとポリシー誘導フロー整流
Outcome-Balanced PPO and Policy-Induced Flow Rectification for Flow-Based VLA Policies
フローベースVLAポリシーの強化学習後学習において、タスク成功率でアドバンテージを再重み付けするPPOと、ポリシーが生成するODE端点ペアに線形ブリッジフローを再フィットする整流を組み合わせ、模擬マニピュレーションで成功率を改善した。
詳しい要約
1. どんなもの?
2. 先行研究と比べてどこがすごい?
3. 技術・手法の肝は?
4. どうやって有効だと検証した?
5. 議論はある?
6. 次に読むべき論文は?
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Yuhan Wang, Ruiyang Wang, Jiaguan Zhu
分類: cs.AI, cs.LG
原文アブストラクト
Reinforcement learning (RL) enables post-training of flow-based vision-language-action (VLA) policies by optimizing expected task return through environment interaction. However, this optimization raises two challenges: policy updates can alter the velocity-field geometry underlying the score approximation used for stochastic transitions, and sparse multi-task rollouts can yield uneven learning signals across tasks and advantage signs. We introduce an RL framework that combines Outcome-Balanced Proximal Policy Optimization (PPO) with policy-induced flow rectification. Outcome-Balanced PPO reweights positive advantages using task success-rate estimates and rescales positive and negative advantages while preserving their signs. Policy-induced rectification periodically fits a linear-bridge flow to ODE endpoint pairs generated by the current policy, refreshing the representation used in subsequent stochastic optimization. We evaluate the framework on simulated manipulation benchmarks and additionally study a component combination in a Real2Sim2Real task, examining task performance, optimization behavior, and flow consistency. The proposed framework improves success rates over the evaluated baselines across the simulated benchmarks.
