より良い方策が監視を難しくする：VLM成功判定におけるカバレッジシフト
When Better Policies Leave Harder Failures: Coverage Shift in VLM Success Monitoring
方策が改善するほど固定VLM判定器が見逃す失敗の種類が変わる「カバレッジシフト」を実験的に示し、絶対見逃し率と条件付き見逃し率・監査カバレッジを併記すべきと提言した論文。
著者: Isabel Guan, Dingyuan Zhang, Shipeng Lyu, I-Ming Chen
分類: cs.RO
原文アブストラクト
Can a better robot policy become harder to monitor? We study a fixed vision-language model (VLM) that judges success while a policy improves. In a controlled 20-seed tray-placement experiment, training selects only simulator-verified successes. Success rises from 66.8% to 97.0%, and accepted hover failures fall from 5.0% to 0.9% of executions. Yet the judge accepts 50.0% of remaining hovers, versus 31.1% under untrained sampling. These residual failures more often lie below the tray rim, which can conceal the gap beneath a grasped cube. A separate boundary probe accepts 148/160 below-rim hovers and 0/160 above-rim hovers. We call this change in conditional monitoring sensitivity coverage shift: learning changes which failures reach a fixed judge. A broader diagnosis covers nine judges and 1,392 stimuli. Judge-filtered experiments across eight settings further show heterogeneous effects of state-dependent acceptance. The results distinguish task improvement from failure detectability and motivate reporting absolute misses, conditional misses, and audit coverage together.