OptRot: Data-Free Rotations Improve LLM Quantization

This paper addresses the challenge of quantizing Large Language Models (LLMs) by introducing a novel method, OptRot, that uses data-free rotations to mitigate weight outliers. This is significant because weight outliers hinder quantization, and efficient quantization is crucial for deploying LLMs on resource-constrained devices. The paper's focus on a data-free approach is particularly noteworthy, as it reduces computational overhead compared to data-dependent methods. The results demonstrate that OptRot outperforms existing methods like Hadamard rotations and more complex data-dependent techniques, especially for weight quantization. The exploration of both data-free and data-dependent variants (OptRot+) provides a nuanced understanding of the trade-offs involved in optimizing for both weight and activation quantization.

• •OptRot is a data-free method for mitigating weight outliers in LLMs.
• •OptRot improves weight quantization performance, outperforming existing methods.
• •OptRot+ incorporates activation covariance for further performance gains.
• •The paper highlights trade-offs between weight and activation quantization in different settings (W4A4 vs W4A8).