ケプラーからニュートンへ：帰納バイアスがTransformerの学習する世界モデルを導く
From Kepler to Newton: Inductive Biases Guide Learned World Models in Transformers
Transformerに空間平滑性・安定性・時間局所性の3つの帰納バイアスを導入することで、惑星軌道の楕円フィッティングを超え、ニュートン力学の力表現を発見できることを示した。
著者: Ziming Liu, Sophia Sanborn, Surya Ganguli, Andreas Tolias
分類: cs.LG, cs.AI, physics.class-ph
原文アブストラクト
Can general-purpose AI architectures go beyond prediction to discover the physical laws governing the universe? True intelligence relies on "world models" -- causal abstractions that allow an agent to not only predict future states but understand the underlying governing dynamics. While previous "AI Physicist" approaches have successfully recovered such laws, they typically rely on strong, domain-specific priors that effectively "bake in" the physics. Conversely, Vafa et al. recently showed that generic Transformers fail to acquire these world models, achieving high predictive accuracy without capturing the underlying physical laws. We bridge this gap by systematically introducing three minimal inductive biases. We show that ensuring spatial smoothness (by formulating prediction as continuous regression) and stability (by training with noisy contexts to mitigate error accumulation) enables generic Transformers to surpass prior failures and learn a coherent Keplerian world model, successfully fitting ellipses to planetary trajectories. However, true physical insight requires a third bias: temporal locality. By restricting the attention window to the immediate past -- imposing the simple assumption that future states depend only on the local state rather than a complex history -- we force the model to abandon curve-fitting and discover Newtonian force representations. Our results demonstrate that simple architectural choices determine whether an AI becomes a curve-fitter or a physicist, marking a critical step toward automated scientific discovery.