動作と言語条件付きビデオ評価による身体化制御のフィードバック
Action- and Language-Conditioned Video Assessment for Embodied Control
視覚ベースの身体化エージェントが複数ステップの指示を実行する際、軌跡全体の進捗を評価する新しい手法ALVAを提案。事前学習済みVLMを用いて、観察・動作系列・指示に基づき離散的な進捗スコアを生成し、閉ループポリシー最適化のフィードバックとして有効性を示した。
著者: Hwanhee Kim, Jaehyun Jang, Seungmin Cha, Hyeonseo Yun, Donghoon Lee, Chang D. Yoo
分類: cs.RO, cs.CV
原文アブストラクト
Vision-based embodied agents executing multi-step natural language instructions require feedback mechanisms that assess task progress over complete trajectories. Conventional approaches based on final-frame matching or continuous embedding similarity may overlook intermediate transitions that are necessary for determining whether an instruction has been completed. We propose ALVA (Action- and Language-Conditioned Video Assessment), a trajectory evaluator that conditions its assessment on visual observations, the executed action sequence, and the natural language instruction. The method uses a pre-trained vision-language model (VLM) in two stages: it first summarizes frame-to-frame visual transitions conditioned on the executed actions and then assesses the generated summary with respect to the instruction to produce a discrete trajectory-level progress score. In simulated 3D household environments, ALVA exhibits a conservative assessment pattern with near-zero false-positive rates. When used as terminal feedback for closed-loop policy optimization, it provides more effective feedback than the evaluated static image and embedding-based visual baselines and reduces the performance gap to a ground-truth oracle. These results support action- and language-conditioned video assessment as an interpretable feedback mechanism for the evaluated simulated embodied-control tasks.