Gemini 4 Carbon: Tops the Index, Hallucinations 4% to 11%

Google announced Gemini 4 Carbon on October 28, two weeks after Gemini 4 Argon reached paying subscribers. The headline is a first place: Carbon scores 59 on version 5.0 of the Artificial Analysis Intelligence Index, one point ahead of Claude Opus 5.5 and two ahead of GPT-6 Astra, according to Google’s own comparison table.

My read: the number to look at sits further down the independent write-ups, and it’s 11%.

What Google says about honesty

In its launch post, Google says Carbon “cuts unsupported claims by approximately 2x over Argon” in internal testing. It also reports that successful cheating on Terminal-Bench 4.0 fell to 0.0%, that high-severity misalignment alerts in its own coding environments dropped by almost 80%, and that the model resisted every attack on Gray Swan’s indirect prompt injection benchmark. Argon had a 0.7% success rate for attackers there.

All of that is Google measuring Google. Mashable makes the same point in its write-up, noting that the safety figures come from Google’s own testing.

What the independent test shows

Artificial Analysis runs a separate test called AA-Omniscience. A summary of its results published on NYU Shanghai’s RITS site says Carbon’s accuracy rose ten points over Argon to 58%. Its hallucination rate also rose, from 4% to 11%.

That’s still the second-lowest of any model measured, behind only Argon. The same summary lists 50% for both Claude Opus 5.5 and GPT-6 Astra. In my read, that makes Carbon far from the sloppiest model measured. But a model that answers more questions correctly and also invents more answers when it doesn’t know is not the same thing as a model that makes fewer unsupported claims.

The two measures may not conflict. Google’s figure comes from its own internal testing, and its post says Carbon puts methodological caveats into the deliverable. Artificial Analysis’ number comes from a separate benchmark, and the summary I read doesn’t explain how it counts a hallucination, so I can’t say what the two have in common. I haven’t opened Artificial Analysis’ own page, only the secondary summary, so check the original before quoting the 4% and 11% yourself.

Where it doesn’t lead

Google’s table has Carbon on top in most rows, and Mashable counts 15 of 17 benchmarks against Argon, GPT-6 Astra, Opus 5.5 and Claude Fable 5.1. The two it doesn’t win: GPT-6 Astra leads on FrontierSWE 2 (65.5% against Carbon’s 61.2%), and Opus 5.5 leads on OSWorld-2.1 (87.2% against 81.5%).

The NYU summary adds that Artificial Analysis puts Carbon second on its separate Coding Agent Index, at 59 against 62 for Claude Opus 5.5 in Claude Code. On the price side, it lists $7.76 per Intelligence Index task for Carbon, about 23% below Opus 5.5’s $10.04 and about 1.2 times GPT-6 Astra’s $6.25.

Benchmark (Google’s table)	Carbon	Argon	GPT-6 Astra	Claude Opus 5.5
AA Intelligence Index v5.0	59	50	57	58
Terminal-Bench 4.0 (AA run)	65.2%	57.1%	59.1%	59.6%
ExploitGym	54.8%	42.0%	42.4%	34.5%
SEC-bench Pro	88.2%	83.4%	85.4%	not listed

One oddity: the NYU summary gives Carbon 64% on Terminal-Bench 4.0 and calls it second place, while Google’s table shows 65.2% and first. I can’t tell from here whether that is a different configuration or a rounding slip, so treat that row as unsettled.

The access problem

You can’t use Carbon. Google says it started giving the model “to a subset of cyber defenders” this week and will widen access through its Fairwind Program, with developers, businesses and consumers coming “as soon as possible”. It gave no date. Google also says it is taking part in the US government’s voluntary pre-release testing process.

Pricing is already public: $2 per million input tokens and $10 per million output tokens, the same introductory rate Argon launched at, with cached input 95% cheaper. Mashable’s view is that announcing early lets Google claim the top spot on the leaderboards now, and gives everyone a look at the benchmarks before the next round of releases. That is Mashable’s read, not a fact. My own take: a price list for a model almost nobody can call is odd, and it means the benchmark claims get little outside scrutiny for now.

For the earlier model, see my post on Gemini 4 Argon’s launch price and its 50% discount.

My read

Google’s cyber numbers are strong on their face. Finding vulnerabilities, checking which ones can be exploited, and writing the patch is a real chain of work, and Google reports leads on all three links. But those external benchmarks were run “without guardrails”, per Google, which says something about the raw model and little about what defenders will get.

On honesty, I’d wait. If Carbon really makes fewer unsupported claims in agent work and more wrong answers on Artificial Analysis’ separate test, which kind of mistake hurts you depends on the job. For a security team reading its output as a patch suggestion, the second kind is probably the one to measure on your own code. We’ll know more when someone outside Google can run it.

Gemini 4 Carbon tops the AI index, and its hallucination rate nearly tripled