MoonshotAI Unveils Kimi-K3: A New Frontier in AI Capabilities
Kimi-K3 Technical Report [pdf]
I am sharing the technical report for Kimi-K3, our latest model designed to push the boundaries of artificial intelligence. This document details our architectural innovations, training methodologies, and performance benchmarks that set Kimi-K3 apart. We aim to provide a transparent look at how we are advancing the field with this new release, offering insights into the future of large language models.
The true measure of an AI's capability lies not just in its size, but in its ability to reason through complex, real-world scenarios with human-like intuition.
- GodelNumbering
Back of the envelope calculation (could be off, correct me if I am)
If you are a large enough company that spends million+ on inference a month, it makes sense to buy a GB300 rack ($6M on top range from what I could find) which has 20.7 TB. Since the model is mixed trained (MXFP4), you would need less than 10% of the rack's memory to serve the full model. Aggregate HBM bandwidth: 576 TB/s. You can run over 6000 parallel agentic workflows (each with ~100k context on average) at ~30 tok/s.
Assuming the annual amortization+electricity at $1.5M/year and about 50% average annual utilization, you get less than 60 cents (USD) per million output token, for a frontier model with plenty of capacity to share, all your data never leaving premises and well over an order of magnitude cheaper!
As long as a company believes that the openweight models will continue to get more capable and 'AI is here to stay', this model provides the first solid footing for a decision to just buy a rack.
- m_ke
Also open sourced a bunch of infra to go with it.
Anyone who claims open source and open weights models are "decel" needs to get their head checked
https://github.com/MoonshotAI/MoonEP
- fahrradflucht
License: https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE
> If the Licensee or any of its affiliates operates a Model as a Service business, and the aggregate revenue of the Licensee and its affiliates exceeds 20 million US dollars (or the equivalent in other currencies) in total over any consecutive 12 months, the Licensee must enter into a separate agreement with Moonshot AI before using the Software or its derivative works for any commercial purpose.
+ the existing 100 million monthly active users, or more than 20 million US dollars for commercial products have to name Kimi clause
- eamag
> we build a self-evolving, hierarchically organized knowledge graph that agents continuously expand through web-scale exploration across knowledge-intensive and coding domains
That's interesting!
- whimsicalism
It's funny that we've finally returned to tanh activation functions, time is a circle.
- a-dub
knowledge graph guided task synthesis. very cool! i have long wondered about the "how do you get good coverage of all the tasks" problem.
maybe some interesting theoretical work there around the rate of production of new knowledge itself and various mechanisms (human approaches, mechanistic approaches, etc).
- storus
What would be the current best method to fine-tune it for my own specific agentic tasks? LoRA + DPO? GRPO? Something else?
- eamag
Can someone explain what are teachers in Multi-Teacher On-Policy Distillation? I can imagine math, coding and other verifiable domains, but they also have biology? Is it where distillation from bigger models come in?