Gradient Dynamics of Attention in Transformers
Research Paper#Transformer Attention, Gradient Descent, Bayesian Inference🔬 Research|Analyzed: Jan 3, 2026 16:27•
Published: Dec 27, 2025 05:31
•1 min read
•ArXivAnalysis
This paper provides a first-order analysis of how cross-entropy training shapes attention scores and value vectors in transformer attention heads. It reveals an 'advantage-based routing law' and a 'responsibility-weighted update' that induce a positive feedback loop, leading to the specialization of queries and values. The work connects optimization (gradient flow) to geometry (Bayesian manifolds) and function (probabilistic reasoning), offering insights into how transformers learn.
Key Takeaways
- •Provides a first-order analysis of attention head dynamics under cross-entropy training.
- •Identifies an 'advantage-based routing law' and a 'responsibility-weighted update'.
- •Shows how these dynamics create a positive feedback loop for query and value specialization.
- •Connects optimization to geometry and function in transformers, explaining how they perform probabilistic reasoning.
Reference / Citation
View Original"The core result is an 'advantage-based routing law' for attention scores and a 'responsibility-weighted update' for values, which together induce a positive feedback loop."