自己中心視点の自然言語クエリ接地のための手の軌跡融合
Hand Trajectory Fusion for Egocentric Natural Language Query Grounding
自己中心視点の長い動画から自然言語クエリに対応する時間区間を特定するタスクで、手の骨格軌跡をエンコードして映像・テキスト特徴と融合する手法を提案し、Ego4Dデータセットで精度向上を確認した。
著者: Enmin Zhong, Carlos R. del-Blanco, Fernando Jaureguizar, Narciso García
分類: cs.CV, cs.AI, cs.HC, eess.IV
原文アブストラクト
Egocentric Natural Language Query (NLQ) grounding asks a model to localize, in a long first-person video, the temporal interval that answers a free-form text query. Existing methods fuse video appearance with the query but ignore hand motion, despite the fact that roughly 41% of Ego4D NLQ queries are answered at a moment of hand--object manipulation or their immediate outcomes.We propose a hand-trajectory encoder for converting a sequence of hand skeletons into highly-semantic hand kinematic features, which are then aligned and combined with pretrained video--text features through a cross-attention fusion strategy with adaptive gating. On the Ego4D NLQ v2 validation split, the clearest gains appear for Hand-Object Interaction queries (+2.54 R1@IoU=0.3) and Quantity/State queries (+4.32 R1@IoU=0.3), indicating that hand trajectory provides grounding cues beyond appearance alone.