SWAN: Memory Optimization for Large Language Model Inference

This research explores a novel method, SWAN, to reduce the memory footprint of large language models during inference by compressing KV-caches. The decompression-free approach is a significant step towards enabling more efficient deployment of LLMs, especially on resource-constrained devices.

• •SWAN optimizes memory usage during LLM inference.
• •The method employs a decompression-free KV-cache compression strategy.
• •This can potentially enable more efficient LLM deployment.