PEFT Methods for RLVR Evaluated

This paper provides a comprehensive evaluation of Parameter-Efficient Fine-Tuning (PEFT) methods within the Reinforcement Learning with Verifiable Rewards (RLVR) framework. It addresses the lack of clarity on the optimal PEFT architecture for RLVR, a crucial area for improving language model reasoning. The study's systematic approach and empirical findings, particularly the challenges to the default use of LoRA and the identification of spectral collapse, offer valuable insights for researchers and practitioners in the field. The paper's contribution lies in its rigorous evaluation and actionable recommendations for selecting PEFT methods in RLVR.

• •DoRA, AdaLoRA, and MiSS are better alternatives to LoRA in RLVR.
• •SVD-informed initialization strategies (PiSSA, MiLoRA) can fail due to spectral collapse.
• •Extreme parameter reduction (VeRA, Rank-1) can severely limit reasoning capacity.