自己注意におけるトークンの動的性質と位置エンコーディングの影響
Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding
事前学習済みTransformerのトークンの連続時間極限のダイナミクスを解析し、トークンが収束・発散する条件を明らかにするとともに、絶対位置・回転位置エンコーディングの影響を調べ、収束を緩和するアーキテクチャ改善を提案した。
著者: Duy-Tung Pham, An The Nguyen, Viet-Hoang Tran, Nhan-Phu Chung, Xin T. Tong, Tan M. Nguyen, Thieu N. Vo
分類: cs.LG
原文アブストラクト
This paper investigates the dynamical properties of tokens in pre-trained Transformer models and explores their application to improving Transformers. To this end, we analyze the dynamical system governing the continuous-time limit of the pre-trained model and characterize the asymptotic behavior of its solutions. Specifically, we characterize when tokens move closer to or farther from one another over time, depending on the model parameters. We provide sufficient conditions, based on these parameters, to identify scenarios where tokens either converge to zero or diverge to infinity. Unlike prior works, our conditions are broader in scope and more applicable to real-world models. Furthermore, we investigate how different forms of positional encoding -- specifically absolute and rotary -- affect these dynamical regimes. Empirical evidence reveals that the convergence scenario adversely impacts model performance. Motivated by these insights, we propose simple refinements to Transformer architectures that mitigate convergence behavior in models with absolute or rotary positional encoding. These findings support theoretical foundations and design principles for improving Transformer models.