音声から直接ロボット操作を指示する音声-言語-行動モデル
Late Breaking Results: Audio-Language-Action Models with Direct Speech Conditioning for Robotic Manipulation
音声をテキストに変換せず直接モータコマンドへ写像するSpeech-π0.5を提案し、ASR経由より3倍低遅延で未知のアクセントにも頑健なことを示した。
著者: Enfu Nan, Pu Zhao, Yixiao Chen, Lin Zhao, Juyi Lin, Chen Wang, Weiwei Chen, Yanzhi Wang
分類: cs.CV
原文アブストラクト
Vision-language-action (VLA) models enable robots to follow instructions in typed text, limiting broader deployment in natural speech. The traditional solution by prepending a speech recognition before VLA to convert speech into texts would incur additional latency and propagate errors. To address this, we present Speech-$π_{0.5}$, which maps raw speech directly to motor commands without explicit textual transcriptions. Specifically, built on $π_{0.5}$, we use a cross-modal projection to convert voice frames directly into audio tokens with significant response latency reduction. To further improve the accuracy, we adopt split LoRA adaptation with dedicated audio and task adapters, under a two-stage training. Speech-$π_{0.5}$ achieves $3\times$ lower response latency than ASR pipeline while maintaining competitive task success on LIBERO, and generalizes robustly to unseen accents where a single adapter degrades significantly.