RollArt: Accelerating Agentic RL Training with Disaggregated Infrastructure

This paper addresses the challenge of efficiently training agentic Reinforcement Learning (RL) models, which are computationally demanding and heterogeneous. It proposes RollArc, a distributed system designed to optimize throughput on disaggregated infrastructure. The core contribution lies in its three principles: hardware-affinity workload mapping, fine-grained asynchrony, and statefulness-aware computation. The paper's significance is in providing a practical solution for scaling agentic RL training, which is crucial for enabling LLMs to perform autonomous decision-making. The results demonstrate significant training time reduction and scalability, validated by training a large MoE model on a large GPU cluster.