ROT: 隠れ状態を文脈ベクトルへ回転させてLVLMの幻覚を抑制
ROT: Rotating Hidden States towards Contextual Vectors for Hallucination Mitigation in LVLMs
大規模視覚言語モデルの中間層で幻覚トークンが示す文脈的逸脱を検出し、ノルムを保つ回転で隠れ状態をマルチモーダル文脈平面へ補正する学習不要の幻覚抑制フレームワークを提案。
詳しい要約
1. どんなもの？
2. 先行研究と比べてどこがすごい？
3. 技術・手法の肝は？
4. どうやって有効だと検証した？
5. 議論はある？
6. 次に読むべき論文は？
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Yijing Du, Xiangcheng Zhan, Shuo Yang
分類: cs.CV, cs.AI, cs.CL
原文アブストラクト
Large Vision-Language Models (LVLMs) frequently suffer from object hallucination. Existing training-free interventions primarily manipulate attention weights, which indirectly affect the deep semantics reaching the final predictive layers. In this work, we shift our focus to the hidden state vectors extracted after self-attention and residual addition. Empirical analysis reveals that hallucinated tokens do not simply over-rely on linguistic priors; instead, they exhibit an anomalous contextual deviation, showing significantly lower similarities to both textual and visual contexts in intermediate layers. Motivated by this, we propose ROT, a layer-specific, training-free framework. ROT dynamically detects semantic deviation in the middle layers and applies a norm-preserving rotation to steer the hidden states back toward the local multimodal context plane spanned by the contexts. For subsequent layers, a representational smoothing mechanism is introduced to stabilize the calibrated trajectory. Extensive experiments on multiple benchmarks demonstrate that ROT consistently reduces hallucinations across various model architectures and scales, offering an efficient, geometry-driven solution for grounded generation.