UK AISI and CAISI Assess Kimi K3's Cyber Capabilities and Exploit Development
UK AISI / Caisi Preliminary Assessment of Kimi K3's Cyber Capabilities
We evaluated Moonshot AI's Kimi K3 model alongside leading U.S. systems using benchmarks like ExploitBench and The Last Ones. While Kimi K3 outperforms previous open-weight models like GLM-5.2, it falls short of frontier U.S. models in complex attack chains. Notably, Kimi K3 failed to achieve arbitrary code execution in our tests, though it did successfully complete a simulated corporate network attack in one out of ten attempts.
Kimi K3 is capable of autonomously attacking small, weakly defended and vulnerable enterprise systems, when directed to do so and given initial network access.
- lebovic
> Kimi K3 performs significantly below the most recent frontier cyber-capable models
UK AISI cyber evals seem to under-elicit capabilities from quirky models [1]. Kimi K3 is a token-hungry model, and I suspect it hit the eval's 100M token limit well before saturating scores [2].
This gap was true for GLM 5.2 as well; they ranked it at Opus 4.5 level [3]. Both anecdotally and with a held-out eval, I've found GLM 5.2 to be better at security research than Opus 4.6 [4]. But it's a quirky model that degrades quickly at long context lengths.
Personally, I'd rank Kimi K3 above Opus 4.8 and lower than GPT 5.6 Sol in its ability to find vulnerabilities and exploit them. But it's not far from the frontier.
[1]: From the UK AISI: "Our setup likely slightly underestimates open weight models’ maximum capability: we didn’t pursue specific elicitation or optimisations which could have improved performance" (https://www.aisi.gov.uk/blog/how-far-behind-the-frontier-are...).
[2]: Their eval also counts cache hits towards the token budget; the 100M token budget is comparable to a ~5M token budget in other evals.
[3]: See GLM 5.2 eval scores in https://www.aisi.gov.uk/blog/how-far-behind-the-frontier-are...
[4]: https://dualuse.dev/posts/chinese-models-are-sometimes-bette...
- gnfargbl
The most important pieces of information in this report are (a) the confirmation that the PRC models have no guardrails and will participate in offensive activity, and (b) the confirmation that they sometimes meet their objectives.
For the purposes of model selection, it's irrelevant to an attacker if a model achieves an offensive objective 70% of the time, when that model refuses to participate 100% of the time. However, a model that always participates but "only" succeeds 10% of the time is golden -- just run it more often, or give it more tokens. Attackers are patient, and many of them are well-resourced.
- hereme888
Is this because cyber capabilities in US models is restricted, so the Chinese can't easily distill those capabilities?
- NitpickLawyer
This again shows the differences in breadth of capabilities that scale offers. On public benchmarks for "regular tasks" the chasing models come close, but on closed ones they lag behind.
Just looking at the Elo differences, k3 is at ~2000 Elo, compared to SotA closed models at 3000 Elo. That is a huge difference. Also, even on the public benchmarks, k3 only scores in the "low hanging fruit tasks", with 0 successful code execution or arbitrary r/rw scenarios.
> Kimi K3 achieved ACE on 0/41 samples, whereas the most cyber-capable models achieved ACE on 20/41 samples on average
But there's hope:
> Kimi K3 reached step 17 of this 32-step attack path on average, while the most cyber-capable U.S. models reached 28.5 steps on average.
> In one of the 10 attempts, Kimi K3 successfully completes “The Last Ones” cyber range within the 100M token limit. This indicates that Kimi K3 is capable of autonomously attacking small, weakly defended and vulnerable enterprise systems, when directed to do so and given initial network access. [...] the most capable models solving it more reliably at 6/10 and 7/10 attempts
The bigger problem is not raw capability, IMO. That can be further RLd into surfacing more reliably. The bigger problem, as seen in the HuggingFace scenario is that SotA models might hit classifiers / guardrails randomly, and leave you with plain refusals. In that case, it is probably better to have something that can help, locally, rather than rolling the dice with API based syste […]
- chvid
It hints that the original Mythos was tuned/trained for cyber attacks.