複雑な組立動作理解のための構成文脈ファインチューニングによる視覚言語モデル
Compositional Context Fine-Tuning Vision-Language Model for Complex Assembly Action Understanding from Videos
組立動作を動詞・対象・工具の要素に分解し、視覚言語モデルを要素ごとにファインチューニングして、ビデオから組立動作を高精度に認識する手法を提案した。
著者: Hao Zheng, Jinyi Huang, Tiantian Zheng, Xun Xu, Tuka Alhanai
分類: cs.CV
原文アブストラクト
Assembly action understanding is a key enabler for effective human-robot collaborative assembly, yet it remains challenging due to subtle motions and fine-grained hand-object interactions. We adapt vision-language models (VLMs) to this challenging domain with Compositional Context Fine-Tuning (CCFT), a method that decomposes assembly actions into semantic elements (Verb, Object, Tool) and fine-tunes VLMs to recognize each action element using templated question-answering pairs. This approach ensures near-deterministic outputs. To enable efficient and effective multi-task learning under limited data, a Layer-Partitioned Alternating Training (LP-AT) method is presented, which assigns distinct model layers to recognize specific action elements through element-specific low-rank adapters. LP-AT alternates weight updates across element-specific adapters, reducing cross-task interference while enabling per-adapter hyperparameter optimization. Furthermore, we create HA-ViD-VQA and IKEA-ASM-VQA datasets from existing assembly video datasets. Extensive experiments on these datasets demonstrate that our method consistently outperforms strong action recognition baselines while providing interpretable element-level predictions that can support diverse downstream applications.