Breaking the Memory Wall for SSMs with Phase Gradient Flow
Analysis
This paper addresses a critical memory bottleneck in the backpropagation of Selective State Space Models (SSMs), which limits their application to large-scale genomic and other long-sequence data. The proposed Phase Gradient Flow (PGF) framework offers a solution by computing exact analytical derivatives directly in the state-space manifold, avoiding the need to store intermediate computational graphs. This results in significant memory savings (O(1) memory complexity) and improved throughput, enabling the analysis of extremely long sequences that were previously infeasible. The stability of PGF, even in stiff ODE regimes, is a key advantage.
Key Takeaways
- •Proposes Phase Gradient Flow (PGF) to overcome memory limitations in SSM backpropagation.
- •PGF achieves O(1) memory complexity, significantly reducing VRAM usage and increasing throughput.
- •Enables sensitivity analysis on extremely long sequences (e.g., chromosome-scale) that were previously infeasible.
- •Maintains stability in stiff ODE regimes, unlike some alternative approaches.
“PGF delivers O(1) memory complexity relative to sequence length, yielding a 94% reduction in peak VRAM and a 23x increase in throughput compared to standard Autograd.”