EMODY Flow: 感情を考慮した音声駆動型全身動作生成
EMODY Flow: Emotion-Aware Audio-Driven Full-Body Motion Generation
音声と感情ラベルから全身のジェスチャーと表情を生成する軽量フローマッチングフレームワークを提案し、感情の識別性を高める補助分類器で従来の感情無視問題を解決した。
著者: Harsh Kumar Agarwal, Xavier Alameda-Pineda, Olivier Perrotin
分類: cs.GR, cs.CV, cs.LG, cs.MM, cs.RO, cs.SD, eess.AS
原文アブストラクト
Embodied conversational agents require synchronized full-body motion (body gestures and facial expressions) that aligns with speech and emotional state. Omni-modal large language models excel at multimodal understanding but produce only linguistic outputs, leaving a critical gap in embodied response generation. We identify and address a failure of emotion conditioning: like other conditional generators that under-use weak conditioning signals, a flow-matching model given both a rich audio embedding and a discrete emotion label suppresses the emotion, generating near-identical motion regardless of the specified emotion. We present EMODY Flow, a lightweight (around 35M parameters) flow-matching framework that attaches to a frozen Qwen-3 Omni model and reuses its internal Mimi audio-codecs to condition two parallel DiT generators - one for SMPL-X body pose, one for FLAME facial expressions. A training-time auxiliary emotion classifier restores emotion sensitivity by forcing generated motion to be emotion-identifiable. EMODY Flow sets a new state of the art on BEAT2 gesture quality, with FGD 0.302, Beat Correlation 0.853, and Diversity 24.62 - improving over the best prior results by 26%, 5%, and 62% respectively - and transfers to zero-shot facial animation on TFHP without domain-specific fine-tuning. Beyond these quantitative gains, the classifier yields clearly emotion-separated motion, which we demonstrate qualitatively through a multidimensional-scaling analysis of the generated gestures.