Google closes the train–inference mismatch gap on TPU with bitwise-identical logprobs

Unlocking Stable On-Policy RL on TPU with Bitwise-Zero Train–Inference Mismatch

Google closes the train–inference mismatch gap on TPU with bitwise-identical logprobs

Reinforcement learning for LLMs can silently go off-policy when the sampler and trainer compute different logprobs for the same action, even under identical weights. Google's Zero-TIM, built in the open-source Tunix library, makes all three paths—sampler decode, serving prefill, and trainer forward—produce bitwise-identical logprobs on a 64-chip TPU v5p mesh. The recipe locks within-shard accumulation, orders cross-shard reductions, standardizes the scoring tail, and wraps serving kernels with custom differentiation rules, yielding zero mean absolute mismatch and higher solve rates on FrozenLake.

The training schedule is on-policy, but the numerical execution has introduced a hidden off-policy gap that compounds over multi-turn trajectories.

More from this day

2026-11-20