論理の幾何学：層別化が意味構造と頑健な推論を生む
The Geometry of Logic: Stratification Induces Semantic Structure and Robust Reasoning
Transformerの内部表現をデータと型の直交部分空間に分離するSTRATを提案し、算術タスクでの分布外誤差を35分の1に削減、11データセットで平均26ポイントの精度向上を達成した。
著者: Cristina V. Lopes, Yuangang Li, Justin Tian Jin Chen, Alberto Krone-Martins, Iris Ma, Md Rakib Hossain Misu
分類: cs.LG, cs.AI
原文アブストラクト
Transformer-based language models perform well on symbolic tasks, yet it remains unclear whether they learn generalizable rules or rely on statistical shortcuts. Mechanistic studies link algorithmic behavior to structured internal representations, motivating the hypothesis that robust reasoning benefits from separating values from the types that control their manipulation. Can making this separation an architectural primitive improve the learnability and generalization of logical mechanisms? We introduce \textbf{STRAT} (\textbf{ST}ratified \textbf{R}egisters \textbf{A}nd \textbf{T}ypes), which partitions the residual stream into orthogonal Data and Type subspaces and uses Type-based attention and gating to govern Data transformations. Controlled arithmetic ablations identify three failure modes associated with data-control interference: the Linear Trap, Gradient Wall, and Open Gate Trap. Mechanistic analysis reveals interpretable logical structure, and in arithmetic, STRAT reduces median OOD error 35-fold relative to a Transformer baseline. On each of 11 datasets spanning 10 tasks, STRAT outperforms the Transformer baseline in mean accuracy, by 26 percentage points on average, with both models trained from 10 base examples per dataset using identical task-specific augmentation where applicable. Under distribution shift, STRAT's mean accuracy drops by only 2.39 percentage points, compared with 11.75 for the Transformer.