Kimi K3 Architecture: A 2.8T Open-Weight Model with Native Multimodal Support
Kimi K3 Architecture Overview and Notes
I break down the Kimi K3 architecture, the largest open-weight model at 2.8T parameters. It scales up the Kimi Linear design with new LatentMoE components and attention residuals for better efficiency. Notably, it replaces all RoPE layers with NoPE and introduces native multimodal support, marking a significant step forward in frontier-level model design.
Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE everywhere instead, making it the first frontier-level architecture to do so.
- Giho
I wonder if Kimis KDA architecture is a way closer to online models(models that learn during interference).As KDA updates its state matrix:
St =St−1 +βt (vt −St−1 kt )ktT
The expression (vt −St−1 kt )ktT is literally the gradient step of a linear regression model updating its weights (S) on a single sample using Mean Squared Error loss.
Interesting what other labs comes up with from this.
- mickael-kerjean
Genuine question: how reproducible / usable / verifiable are these architectures from the published documentation? Are they similar to PDF/DWG/PSD specifications, where the format look like an open spec at first sight until you attempt to implement it and realize the crucial implementation details are undocumented?
- thatsgcasey
Sabastian Raschka is one of the great LLM researchers/authors. I highly recommend his substack
- augment_me
I feel like the Kimi team is amongst the best in the industry to pick and choose what is meaningful from the other models. For example, avoiding the expensive and empirically uncertain mHC in favor of simpler residuals. Latent MoE.
My only doubts are around Linear Attention instead of DSA as this is inherently lossy. You are kind of banking on that your query is inherently in the embedding space of the model already and can be lossy.
- madhu_ghalame
A few diagrams showing how the data flows through the architecture would make these concepts much easier to grasp.
- gboss
Anybody getting the result that Kimi 3 is more expensive than Opus 5 or Sol on Cursor? Pretty sure Kimi 3 sucked up a good chunk of my ultimate plan in a few prompts. Anyone have any tools or ways to understand per model usage towards cursor subscriptions? I know there are alternatives to cursor just haven’t made the move yet. (Edit spelling)
- Ilaurens
"Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead."
It just baffles me that this even works at all. Doesn't it just become a token soup? Is attention that precise that a second token can tell its the second token just because it learns to accumulate something in the embedding space without any sort of inductive bias?
- dools
They may have all this amazing architecture but Kimi has been super dumb recently. I reckon they’re under compute pressure and quantising to stay afloat. I was a heavy k2.5/2.6 user earlier in the year and built serious features with it, but even k3 now does stupid shit like fail tool calls and get stuck in endless thought trains. K2.6 was spinning its wheels on a problem for over 10 minutes today and then deepseek v4 fixed it in under a minute.
Something is wrong at moonshot.