表現には軌跡がある：ニューラルネットワークにおける時間的表現ダイナミクスの特性評価
Representations Have Trajectories: Characterizing Temporal Representation Dynamics in Neural Networks
訓練中の表現幾何学を大規模に追跡し、ネットワークが高ランク持続・崩壊と回復・低ランク持続という3つの幾何学的軌跡をたどること、および回復不能な崩壊（DRC）が表現能力を永続的に失わせる病理的状態であることを示した研究。
著者: Shivaprasad B. Gowda, Sparsh Mittal
分類: cs.LG
原文アブストラクト
Recent work has shown that deep neural networks exhibit an implicit low-rank simplicity bias. In this paper, we investigate the questions that naturally arise from this observation. What are the temporal consequences of low-rank simplicity bias during optimization? How does representation geometry evolve throughout optimization rather than only at convergence? Do different neural architectures exhibit distinct geometric optimization behaviors? To answer these questions, we conduct a large-scale empirical study tracking representational geometry throughout training. We find that neural networks follow three recurring geometric trajectories: persistent high-rank, collapse-and-recovery, and persistent low-rank. We further identify Destructive Representation Collapse (DRC) as a pathological regime in which representations collapse to rank-one and fail to recover, leading to a sustained loss of representational capacity. While collapse-and-recovery is consistently associated with successful optimization in controlled settings, experiments on real-world models show that the relationship between geometric trajectories and optimization outcomes is configuration dependent. Overall, our findings suggest that representation geometry evolves dynamically during training and provide a new perspective on how deep neural networks learn.