Continual Learning for LLMs: Merge Before Forgetting with LoRA
Published:Dec 28, 2025 17:37
•1 min read
•ArXiv
Analysis
This paper addresses the challenge of catastrophic forgetting in large language models (LLMs) within a continual learning setting. It proposes a novel method that merges Low-Rank Adaptation (LoRA) modules sequentially into a single unified LoRA, aiming to improve memory efficiency and reduce task interference. The core innovation lies in orthogonal initialization and a time-aware scaling mechanism for merging LoRAs. This approach is particularly relevant because it tackles the growing computational and memory demands of existing LoRA-based continual learning methods.
Key Takeaways
- •Proposes a novel continual learning method for LLMs using LoRA.
- •Employs orthogonal initialization and time-aware scaling for merging LoRAs.
- •Aims to improve memory efficiency and reduce task interference.
- •Maintains constant memory complexity with respect to the number of tasks.
Reference
“The method leverages orthogonal basis extraction from previously learned LoRA to initialize the learning of new tasks, further exploits the intrinsic asymmetry property of LoRA components by using a time-aware scaling mechanism to balance new and old knowledge during continual merging.”