G-KV: Optimizing LLM Inference with Decoding-Time KV Cache Eviction

This research explores a novel approach to enhance Large Language Model (LLM) inference efficiency by strategically managing the Key-Value (KV) cache during the decoding phase. The paper's contribution lies in its proposed method for KV cache eviction utilizing global attention mechanisms.