MoRAgent: Parameter-Efficient Agent Tuning with Mixture-of-Roles

This paper addresses the challenge of parameter-efficient fine-tuning (PEFT) for agent tasks using large language models (LLMs). It introduces a novel Mixture-of-Roles (MoR) framework, decomposing agent capabilities into reasoner, executor, and summarizer roles, each handled by a specialized Low-Rank Adaptation (LoRA) group. This approach aims to reduce the computational cost of fine-tuning while maintaining performance. The paper's significance lies in its exploration of PEFT techniques specifically tailored for agent architectures, a relatively under-explored area. The multi-role data generation pipeline and experimental validation on various LLMs and benchmarks further strengthen its contribution.