Zero Mismatch, Simple Async RL: REINFORCE Zero, DeepSeek-V4.1 and Kimi K3.1

Asynchronous RL is faster, but it trains on samples generated by older weights. The rollout-to-learner gap has two parts. Weight lag changes smoothly and unclipped importance sampling (IS) controls it. Kernel mismatch, where rollout and training compute different numbers for the same weights, fluctuates and persists despite IS. Align the kernels, and unclipped IS corrects the remaining weight lag without sample-rejection gates. REINFORCE Zero (NVIDIA Research and Inferact) trains stably for 20,000 updates, learns at policy lag 64, and reaches 2.40× the training-step throughput of synchronous training with aligned kernels. Zero mismatch now covers DeepSeek-V4.1-Flash (DAPO rollout step within 10% of the fast baseline), Kimi K3.1 on day 0, and both DeepSeek-V4-Flash and Kimi K3.1 on AMD MI355X. Everything is in Inferact/zero-mismatch-rl. 1. From synchronous to asynchronous Last month, we showed that rollout and training can agree exactly. On DeepSeek-V4-Flash, vLLM and the trainer produced the same logprob for every sampled token, bit for bit, on the fast kernels, with an 8.7% reduction in rollout throughput. An agent skill now does the alignment…