The 4-Bitter Lesson: Balancing Stability and Performance in NVFP4 RL

We explore the delicate tradeoff between training speed and stability when using NVFP4 precision for Reinforcement Learning. While low-precision formats like NVFP4 on NVIDIA Rubin GPUs offer massive throughput gains, they introduce policy drift that can crash training. Our new recipe combines selective high-precision layers and dequantized backward passes to maintain stability, enabling efficient long-horizon RL without sacrificing model performance.

"The essence of RL is to teach a model through action and consequence; at humans&, we use RL to train models that understand the long-term impacts of their interactions with people."