価値モデル評価2026/10/19arXiv:2610.26663

RoboValue：ロボット価値モデルの統合評価のための細粒度シミュレーション・実世界ベンチマーク

RoboValue: A Fine-Grained Sim-and-Real Benchmark for Unified Evaluation of Robotic Value Models

ロボット価値モデルをタスク状態理解・時間的進捗監視・失敗回復推論・価値一貫性の4次元で統合評価する、シミュレーションと実世界にまたがる細粒度ベンチマークを提案した論文。

詳しい要約

1. どんなもの？

- RoboValueはロボットのvalue modelを統一的に評価するfine-grainedなsim-and-real benchmark - 視覚観測とinstructionからtask実行を評価するvalue modelの信頼性を測る枠組み - 共有interfaceで4つの相補的能力次元を評価 - task-state understanding - temporal progress monitoring - failure and recovery reasoning - value consistency - RoboValue-Datasetを提供 - 15 simulationと20 real-world manipulation tasks - 訓練用expert demonstrationsとfine-grained評価用annotated trajectories - public leaderboardとwebsiteを提供

2. 先行研究と比べてどこがすごい？

- 既存評価はexecution understandingの一部側面のみを対象 - value feedbackの信頼性が十分に検証されていなかった - RoboValueは4能力次元をshared interfacesで統合評価 - diagnostic trajectoriesとinstruction variationsでtask要求・実行変化への反映を検証 - simとrealの両方をカバーし、embodiment・environmentのshiftも分離評価 - 具体的なvalue supervisionの改善対象を特定

3. 技術・手法の肝は？

- 4つのcapability dimensionsをshared interfacesで評価するframework - task-state understanding - temporal progress monitoring - failure and recovery reasoning - value consistency - diagnostic trajectoriesとinstruction variationsを活用 - value judgmentがtask requirementsや実行中の変化を正確に反映するか検査 - RoboValue-Dataset - 15 simulation・20 real-world manipulation tasks - expert demonstrationsでtraining - annotated trajectoriesでfine-grained evaluation - zero-shotとone-shot設定で代表value modelを評価 - standard conditi…

4. どうやって有効だと検証した？

- 代表的なrobotic value modelsをzero-shot・one-shotで評価 - standard conditionsに加えrobot embodimentとenvironmentのshiftを分離して検証 - 結果として以下を確認 - fine-grained instruction groundingの誤り - history-dependent progress trackingの誤り - recovery assessmentの誤り - outcomeやforward-progress scoresが強いmodelでも生じる - 同じsemantic subtaskでも有効なexecution order間でvalue gainsが大きく異なる - datasetとpublic leaderboardで継続評価を支援

5. 議論はある？

- fine-grained instruction groundingの誤りが示唆される - history-dependent progress trackingの課題 - recovery assessmentの課題 - outcome・forward-progress scoresが高くても上記誤りが残る - 同じsemantic subtaskのvalue gainsがvalid execution orders間で大きく異なる - これらはvalue supervisionの具体的改善対象 - ただし要旨からは不明な点も多い - 各誤りの定量的大小 - model間の詳細比較 - real-world評価の詳細条件

6. 次に読むべき論文は？

- 要旨で参照・比較されている具体的なvalue model名は不明 - 関連手法としてrobotic value models、policy learningへのfeedback、execution monitoringを挙げる - 同分野の定番としてimitation learning、reinforcement learning、vision-language modelsを用いたvalue評価、sim-to-real transferの文献を次に読むべき - RoboValueのwebsite・public leaderboardで評価対象modelを確認する

※ AIが要旨から生成した要約です。正確性は原文をご確認ください。

著者: Shengbang Liu, Zhengye Du, Zhilong Wan, Honghao Su, Chenxiang Xia, Chang Ge, Jinyang Xiao, Nan Wang, Zhipeng Hu, Huining Yuan, Shu'ang Yu, Yue Chen, Tianxing Chen, Chuankang Li, Maoqing Yao, Wenbo Ding, Yu Wang, Guanghui Ren, Chao Yu

分類: cs.RO

原文アブストラクト

Robotic value models assess task execution from visual observations and instructions, providing feedback for policy learning and execution monitoring. However, existing evaluations cover only selected aspects of execution understanding, leaving the reliability of value feedback insufficiently tested. We introduce RoboValue, a fine-grained sim-and-real benchmark for unified evaluation of robotic value models. Its framework evaluates four complementary capability dimensions through shared interfaces: task-state understanding, temporal progress monitoring, failure and recovery reasoning, and value consistency. Diagnostic trajectories and instruction variations test whether value judgments accurately reflect task requirements and changes throughout execution. RoboValue-Dataset covers 15 simulation and 20 real-world manipulation tasks, providing expert demonstrations for training and annotated trajectories for fine-grained evaluation. We evaluate representative robotic value models in zero-shot and one-shot settings under standard conditions and separate shifts in robot embodiment and environment. Our results reveal errors in fine-grained instruction grounding, history-dependent progress tracking, and recovery assessment, even in models with strong outcome and forward-progress scores. The same semantic subtask can also receive substantially different value gains across valid execution orders. These findings identify concrete targets for value supervision, while the dataset and public leaderboard support continued assessment of feedback reliability. The website is available at https://robovalue-benchmark.github.io/.

arXivで原文を読む →arXiv概要ページ

← 論文一覧