IsoExec: Unified Execution to Eliminate Trainer-Inference Mismatch in SkyRL

In theory, on-policy RL assumes that rollout and training evaluate the same policy. In practice, RL training systems often use two separate engines for rollout and training, with different model definitions, kernels, batch shapes, and parallelism layouts. Because floating-point arithmetic is non-associative, those differences can change token probabilities even when executing the same policy. This can make new RL algorithms, changes in harnesses and environments, and improvements to RL infrastructure and hardware kernels hard to debug. To resolve this problem, we introduce IsoExec, a cross-framework unified execution abstraction for eliminating mismatch between the training and inference engines for RL workloads. IsoExec has two components: an execution contract that specifies and enforces the execution details affecting floating-point rounding across engines, and a unified model with aligned, batch-invariant kernels that are bitwise consistent across training and rollout. We implement IsoExec in SkyRL with vLLM and Megatron. On a single 8×H100 node, with synchronous Qwen3.5-35B-A3B DAPO training, we reduced the average end-to-end rollout-versus-training logprob difference below , with 25% overhead compared with the current SkyRL baseline over 50 steps.…