analiza aiqbench
LongStraw addresses a critical bottleneck in AI agent development by enabling Reinforcement Learning post-training on contexts exceeding 2 million tokens within fixed GPU budgets. The system achieves this through novel techniques like response replay and distributed state management that eliminate the need to retain full computational graphs during backpropagation. This advancement is essential for training agents capable of processing long trajectories involving complex observations, tool outputs, and decision histories without prohibitive memory costs.
LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget
arXiv:2607.14952v2 Announce Type: replace Abstract: A widening gap separates million-token inference from RL post-training, which remains at 256K tokens or below. The gap matters for AI agents, whose observations, tool outputs, documents, and decisions accumulate over long trajectories. Unlike inference, GRPO scores and backpropagates through multiple responses conditioned on one history, making attention and long-lived backward state a primary GPU-memory barrier. We present LongStraw, an objective- and architecture-aware system for million-token RL post-training under a fixed GPU budget. Resident state retains only model-native prompt state needed by later tokens, not the full graph. Response replay restores that state, scores graph-free old/reference branches, rebuilds one policy response under autograd, backpropagates, and pops to the prompt boundary. Distributed, model-native execution assigns state and gradients to context and expert owners. State is shared within a group and recaptured or reused under a measured refresh policy after an update. For Qwen3.6-27B, LongStraw combines compact GDN state, CP8-sharded KV pages, exact attention composition, and reverse block replay. For GLM-5.2, it combines CPU-resident MLA/DSA state, IndexShare-aware selection, and native top-8 MoE replay over CP32/EP32. On eight H20 GPUs, Qwen completes exact-attention response-only GRPO at 2,097,152 positions for G=2 and G=8; a 4,456,448-position prefix supports eight G=8 cycles (64 replays) at 83.894 GB per rank. On 32 H20 GPUs, GLM completes deterministic 2M execution and two 78-layer backward passes; archived external integrations provide preliminary validation of the real vLLM-DAPO-Tinker/Megatron loop. Practical training context is set by resident-state lifetime, replay, and distributed ownership rather than attention kernels alone. The measured objective is response-only execution, not full-sequence gradient equivalence.
Abstract:A widening gap separates million-token inference from RL post-training, which remains at 256K tokens or below. The gap matters for AI agents, whose observations, tool outputs, documents, and decisions accumulate over long trajectories. Unlike inference, GRPO scores and backpropagates through multiple responses conditioned on one history, making attention and long-lived backward state a primary GPU-memory barrier. We present LongStraw, an objective- and architecture-aware system for million-token RL post-training under a fixed GPU budget. Resident state retains only model-native prompt state needed by later tokens, not the full graph. Response replay restores that state, scores graph-free old/reference branches, rebuilds one policy response under autograd, backpropagates, and pops to the prompt boundary. Distributed, model-native execution assigns state and gradients to context and expert owners. State is shared within a group and recaptured or reused under a measured refresh policy after an update. For Qwen3.6-27B, LongStraw combines compact GDN state, CP8-sharded KV pages, exact attention composition, and reverse block replay. For GLM-5.2, it combines CPU-resident MLA/DSA state, IndexShare-aware selection, and native top-8 MoE replay over CP32/EP32. On eight H20 GPUs, Qwen completes exact-attention response-only GRPO at 2,097,152 positions for G=2 and G=8; a 4,456,448-position prefix supports eight G=8 cycles (64 replays) at 83.894 GB per rank. On 32 H20 GPUs, GLM completes deterministic 2M execution and two 78-layer backward passes; archived external integrations provide preliminary validation of the real vLLM-DAPO-Tinker/Megatron loop. Practical training context is set by resident-state lifetime, replay, and distributed ownership rather than attention kernels alone. The measured objective is response-only execution, not full-sequence gradient equivalence.
46 pages, 10 figures, 11 tables. Code: this https URL
Machine Learning (cs.LG); Distributed, Parallel, and Cluster Computing (cs.DC)
arXiv:2607.14952 [cs.LG]
(or arXiv:2607.14952v2 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 282 cuvinte
Subiecte conexe
DI Series: Serving Kimi-K3 MXFP4 Wide-EP Disaggregated on AMD Instinct MI300X and MI325X
AMD · 8 oct. 2026
[AINews] not much happened today
Latent Space · 3 oct. 2026
Can we change an AI model’s architecture without retraining it from scratch?
Amazon Science · 12 oct. 2026
[AINews] Opus 5.5 is good at explainer videos
Latent Space · 29 sept. 2026