Efficient Long-Context Attention
Published:Dec 30, 2025 03:39
•1 min read
•ArXiv
Analysis
This paper introduces LongCat ZigZag Attention (LoZA), a sparse attention mechanism designed to improve the efficiency of long-context models. The key contribution is the ability to transform existing full-attention models into sparse versions, leading to speed-ups in both prefill and decode phases, particularly relevant for retrieval-augmented generation and tool-integrated reasoning. The claim of processing up to 1 million tokens is significant.
Key Takeaways
- •Introduces LongCat ZigZag Attention (LoZA) for sparse attention.
- •Enables speed-ups in long-context scenarios.
- •Applicable to prefill and decode phases.
- •Claims processing up to 1 million tokens.
Reference
“LoZA can achieve significant speed-ups both for prefill-intensive (e.g., retrieval-augmented generation) and decode-intensive (e.g., tool-integrated reasoning) cases.”