Rethinking Fine-Tuning for Vision-Language Models

This paper introduces Mask Fine-Tuning (MFT) as a novel approach to fine-tuning Vision-Language Models (VLMs). Instead of updating weights, MFT reparameterizes the model by assigning learnable gating scores, allowing the model to reorganize its internal subnetworks. The key contribution is demonstrating that MFT can outperform traditional methods like LoRA and even full fine-tuning, achieving high performance without altering the frozen backbone. This suggests that effective adaptation can be achieved by re-establishing connections within the model's existing knowledge, offering a more efficient and potentially less destructive fine-tuning strategy.