Moonshot's new Kimi scores 81.0% on its own hacking test. An unfiltered version scored 90.5%.
Kimi K3.1 Pro is on the API now, with full weights promised by November 4. Moonshot says it stripped cyber training data first. The benchmark it used to prove that is its own.
Moonshot AI announced Kimi K3.1 Pro on October 24, per Reuters, and put one number in the middle of it: 81.0%. That’s the share of hard exploit-writing tasks the model solved on the company’s own test, counting three attempts per task. A checkpoint from the same pipeline, trained without the cyber filtering, solved 90.5%.
The gap is nine and a half points, and Moonshot published that before-and-after itself for a model it plans to give away.
What Moonshot says it did
According to Moonshot’s announcement, the team noticed vulnerability-research skill rising steadily as post-training scaled up. When they traced it, they found cyber-related material sitting inside the general mid-training data, and cyber tasks mixed into the reinforcement-learning environments. Neither was put there on purpose. It came along with broad data.
So they removed the cyber content from both places and trained again. The test is called xdev-bench-hard. Its tasks mostly reproduce recently disclosed bugs in software like the Linux kernel, Node.js, JavaScriptCore and Exim, with mitigations switched on. The model gets source code and has to turn a flaw into working code execution.
The result, at pass@3 (three attempts per problem):
| Model | xdev-bench-hard |
|---|---|
| Same pipeline, no cyber filtering | 90.5% |
| Released K3.1 Pro | 81.0% |
Moonshot adds that K3.1 Pro still improves a lot over its predecessor, K3, which it reads as general coding skill carrying over into security work.
The test is Moonshot’s own
No outside group has checked that table. Reuters, in its October 24 report, notes the results haven’t been independently verified. The benchmark was built in-house, and Moonshot hasn’t said it will release the 90.5% checkpoint, so nobody outside can rerun that comparison.
Nine points off a benchmark in a lab sandbox doesn’t tell you how much harder a real intrusion got. In my view, writing an exploit for a known bug with the source in hand is a narrow slice of what attackers do.
Why it’s a big model release anyway
Strip the safety angle away and this is also a strong model at a low price. Simon Willison, who saw it on the API on October 23, notes that the API costs $2 per million input tokens and $10 per million output tokens, the same as GPT-6.1 Sol and half the price of Claude Opus 5.5. A faster Highspeed version runs at twice the price. Artificial Analysis, as Willison reports it, scored K3.1 Pro at 55. Claude Fable 5.1, GPT-6 Astra and Gemini 4 Argon sit at 53, and only Claude Opus 5.5 is higher, at 58.
Moonshot’s own charts put it roughly level with Opus 5.5 and GPT-6 Astra. Treat that one as vendor marketing until people have run it.
On October 6 I wrote about Reflection’s Beam, pitched as America’s open-model answer, whose own chart had it behind four Chinese models. If the Artificial Analysis score holds, Kimi is now the one setting the open-weight bar.
What an open release changes
A hosted model can be rate-limited, monitored and patched. Weights can’t. Once the files are out, anyone can fine-tune them, and Reuters points out that the original developer then has less ability to restrict access or update safeguards.
That’s why training-time removal is a more interesting move than a content filter. My guess is that it’s harder to undo, though not impossible. Fine-tuning on exploit data could plausibly win back some of the lost ground, and the announcement doesn’t say how hard that would be. I’d like to see someone test it after November 4.
The company states its policy in two parts. It doesn’t want to “extend the absolute frontier of cyber capability” with open weights, and it intends to raise the cyber ability in later open releases over time. It also argues that open models help defenders audit old software that nobody is looking after. The Tower Post’s write-up adds that the benchmark runs use a sample of tasks with no hard timeouts or output-token limits, so the permissive setup shouldn’t be read as odds against a defended network.
There’s a political layer too. On October 16 China published draft rules that would restrict exports of AI-driven malware tools and vulnerability-discovery software, which I wrote about in our post on the export-control draft. That draft is open for public comment. Whether Moonshot’s filtering was shaped by it, the sources don’t say, and I won’t guess.
My read
I think the disclosure is more useful than the number. A lab telling you which benchmark it lowered and by how much is rare, and it gives researchers something specific to test. But “we reduced it” is a claim about a training run, not a guarantee about a model that thousands of people will modify.
The weights are a promise until they exist. If they show up by November 4, what I want to know is whether independent exploit benchmarks land near 81% or somewhere else. On outside testing, there’s one earlier data point. Mindgard, a UK security firm, says it jailbroke two older Kimi models, and says Moonshot only replied after the BBC asked.
I read Moonshot’s post, Reuters and Willison in full, and used The Tower Post for the benchmark-setup caveat. The China draft and Mindgard details come from my earlier posts, linked above, which carry their own sources.