Kimi K3 Architecture: A 2.8T Open-Weight Model with Native Multimodal Support
Kimi K3 Architecture Overview and Notes
I break down the Kimi K3 architecture, the largest open-weight model at 2.8T parameters. It scales up the Kimi Linear design with new LatentMoE components and attention residuals for better efficiency. Notably, it replaces all RoPE layers with NoPE and introduces native multimodal support, marking a significant step forward in frontier-level model design.
Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE everywhere instead, making it the first frontier-level architecture to do so.