FRoD: Efficient Fine-Tuning for Faster Convergence

This paper introduces FRoD, a novel fine-tuning method that aims to improve the efficiency and convergence speed of adapting large language models to downstream tasks. It addresses the limitations of existing Parameter-Efficient Fine-Tuning (PEFT) methods, such as LoRA, which often struggle with slow convergence and limited adaptation capacity due to low-rank constraints. FRoD's approach, combining hierarchical joint decomposition with rotational degrees of freedom, allows for full-rank updates with a small number of trainable parameters, leading to improved performance and faster training.

• •FRoD is a novel fine-tuning method for large language models.
• •It aims to improve convergence speed and efficiency compared to existing PEFT methods.
• •FRoD achieves performance comparable to full model fine-tuning with significantly fewer trainable parameters.
• •The method combines hierarchical joint decomposition with rotational degrees of freedom.