analiza aiqbench
This paper addresses the systems side of reinforcement learning for LLMs: how completed rollouts move to the training stage in disaggregated pipelines. It claims up to 93% lower architectural staleness versus train-batch admission and up to 5.8x speedup in transfer plus training. These are results from the authors' own experiments, and it's unclear how they generalize beyond their workloads. For ML practitioners, the useful question is whether the staleness-versus-efficiency tradeoff holds in production-scale RL training.
KnitTrans: Coordinated Trajectory Transfer for Effective and Efficient LLM Reinforcement Learning
De la arXiv
Text integralarXiv:2610.32389v1 Announce Type: cross Abstract: Trajectory transfer from rollout to training affects both learning quality and system efficiency in large language model reinforcement learning (RL). Enforcing batch-wide training requirements can delay the admission of completed trajectories, resulting in higher architectural staleness: the additional parameter-version gap from disaggregated pipelining relative to colocated execution. Beyond admission, misalignment between data access and training consumption causes redundant movement and unnecessary waiting when transferring large trajectory payloads.
We present KnitTrans, a system that coordinates the rollout-to-training path through three designs. First, constraint-preserving mini-batch admission allows flexibility in early mini-batch composition, while checking that enough capacity remains to satisfy the remaining quotas. Second, on-demand trajectory data access matches storage units to consumer assignments and overlaps micro-batch data preparation with computation. Third, multimodal encoding deduplication supports reuse across a mini-batch while preserving gradient contributions from all backbone micro-batches. Experiments show up to 93% lower architectural staleness than with train-batch admission, narrowing the learning-quality gap between disaggregated execution and colocated execution.
On multimodal workloads, KnitTrans reduces transfer latency by two orders of magnitude and achieves a speedup of up to 5.8 times in transfer plus training over the baseline.
Abstract:Trajectory transfer from rollout to training affects both learning quality and system efficiency in large language model reinforcement learning (RL). Enforcing batch-wide training requirements can delay the admission of completed trajectories, resulting in higher architectural staleness: the additional parameter-version gap from disaggregated pipelining relative to colocated execution. Beyond admission, misalignment between data access and training consumption causes redundant movement and unnecessary waiting when transferring large trajectory payloads. We present KnitTrans, a system that coordinates the rollout-to-training path through three designs.
First, constraint-preserving mini-batch admission allows flexibility in early mini-batch composition, while checking that enough capacity remains to satisfy the remaining quotas. Second, on-demand trajectory data access matches storage units to consumer assignments and overlaps micro-batch data preparation with computation. Third, multimodal encoding deduplication supports reuse across a mini-batch while preserving gradient contributions from all backbone micro-batches. Experiments show up to 93% lower architectural staleness than with train-batch admission, narrowing the learning-quality gap between disaggregated execution and colocated execution.
On multimodal workloads, KnitTrans reduces transfer latency by two orders of magnitude and achieves a speedup of up to 5.8 times in transfer plus training over the baseline.
Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG)
arXiv:2610.32389 [cs.DC]
(or arXiv:2610.32389v1 [cs.DC] for this version)
https://doi.org/10.48550/arXiv.2610.32389 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 286 cuvinte
Subiecte conexe
Sampling-Based Coordination-Informed Multi-Objective Multi-Robot Reinforcement Learning
arXiv · 17 aug. 2026
CRAFT: Coaching Reinforcement Learning Autonomously using Foundation Models for Multi-Robot Coordination Tasks
arXiv · 8 iun. 2026
Hierarchical Multi-agent Reinforcement Learning for Warehouse Robot Coordination under Communication Loss
arXiv · 29 sept. 2026
Contract-Based Compositional Shielding for Safe Multi-Agent Reinforcement Learning
arXiv · 15 iun. 2026