FailBench: VLMはロボットタスクの成功判定をどこまで信頼できるか？
FailBench: How Reliable are VLMs at Judging Robot Task Success?
ロボット操作の失敗検出を評価するベンチマークFailBenchを構築し、13種類のVLMベース検出器の性能を分析。最高性能でも平均バランス精度0.77にとどまり、接触を伴う組立タスクではほぼ偶然レベルに低下することを示した。
著者: Zaruhi Navasardyan, Tatul Danielyan, Hrant Davtyan
分類: cs.RO, cs.AI
原文アブストラクト
Vision-Language Models (VLMs) are increasingly used to evaluate robot manipulation outcomes, but existing benchmarks offer limited evidence of cross-domain generalization. We introduce FailBench, a benchmark for robot failure detection comprising 2,197 manipulation attempts across 14 public sources (12 real-world, 2 simulated). In FailBench, 75% of failures occur naturally, and six real-world sources come from non-failure-detection datasets. Evaluating 13 VLM-based detectors, we find the best model achieves only 0.77 mean balanced accuracy. Notably, models fine-tuned for failure detection consistently underperform general-purpose VLMs and their own pretrained baselines. Performance depends heavily on required visual evidence: models approach saturation when outcomes depend on observable object motion, but degrade to near-chance (<0.60 balanced accuracy) on contact-intensive assembly tasks. Error analysis reveals a systematic bias toward predicting success under ambiguous evidence, which persists even with increased reasoning effort. Finally, we show that input-level intervention--spatially localizing and cropping outcome-relevant regions--improves the top detector by 2.4 percentage points without extra training.