GPT-6 Astra Computer Use: Shield's Real Test

OpenAI published a case study on November 13 about Shield, an IT support platform, and its GPT‑6 Astra computer-use model that shipped in September. The case study says Shield now hands browser-based IT tickets to agents instead of leaving a technician to finish them by hand. I read that case study plus OpenAI’s own launch benchmarks for Astra and an independent benchmark tracker, and they don’t quite agree on how good “the best computer-use model” really is.

Mayank Jain runs technology at Shield. His agents kept hitting the same wall: old portals. Password resets, license lookups, device checks, the stuff a technician does forty times a day, almost none of it lives behind an API. An agent that can’t click through a 2012-era web form just sits there, and the ticket waits for a human to log in and finish it by hand.

What Shield says changed

Jain told OpenAI his team tried computer use before Astra. It wasn’t good enough. Those earlier agents could read the screen, move the cursor and type like a person would, but they stalled on old interfaces more than they got through them. “Usually, a system that doesn’t have an API is an old-school system that also doesn’t look very nice and is very hard to use, even by a human,” he said.

With Astra, Jain says Forge’s agents now complete browser tasks end to end, thousands of runs a month across Shield’s IT-provider customers, with fewer failures and timeouts than before. Technicians still decide which actions an agent can take unsupervised and where it needs sign-off. Phone calls, on-site visits and remote-desktop sessions aren’t handed to agents yet; Jain says those are still in testing. Customer satisfaction on AI-assisted tickets is higher, he says, though OpenAI’s write-up doesn’t publish a number for that one.

None of this is independently verified.

It’s one customer’s account, written up by the vendor that sold them the model, and every number in it is Shield’s or OpenAI’s to check. I’m including it because it’s a real deployment with a specific before-and-after, not because a company’s own case study is neutral evidence.

The claim behind it

OpenAI’s pitch for Astra isn’t just “it’s better.” The launch post for GPT-6 Astra calls it “the world’s best computer use model” and backs that with a timing number: on OSWorld 2.0, Astra scores 72.6% in roughly 40 minutes per task, against 65.7% in about 75 minutes for its predecessor, GPT‑5.6 Sol. That’s the number Shield’s story is really leaning on. A technician-grade agent that’s twice as fast and still more accurate is the difference between “the agent finishes the ticket” and “the agent gets stuck and a human finishes it anyway.”

My read: that speed claim is the part of this story that matters more than the case study. A single IT vendor’s workflow numbers tell you Astra works well enough for one company’s use case. A benchmark that measures speed and accuracy together tells you something closer to why.

Where “world’s best” gets complicated

Here’s the part OpenAI’s launch post doesn’t put next to the 72.6%: on the independent benchmark tracker benchlm.ai’s OSWorld 2.0 leaderboard, which collects vendor-reported scores on the same test, Moonshot AI’s Kimi K3.1 Pro is listed at 79.6%, ahead of Astra’s 72.6%. Both numbers are self-reported by their own labs, run under what the tracker calls comparable protocol, and the tracker is upfront that different releases of OSWorld 2.0 aren’t always directly comparable to each other.

So “the world’s best computer use model” is true on OpenAI’s own speed-adjusted comparison against its own prior model, and not true against at least one other lab’s score on the same public benchmark. Both things can be accurate at once: Astra’s efficiency gain over GPT‑5.6 Sol is real, and a different model still posts a higher raw OSWorld score. OpenAI picked the comparison that makes Astra look best, which is what every vendor’s launch post does. It’s worth checking the chart, not just the pull quote, before taking “best in the world” at face value.

Model	OSWorld 2.0 score	Source
Kimi K3.1 Pro (Moonshot AI)	79.6%	Reported by Moonshot AI
GPT-6 Astra (OpenAI)	72.6%	Reported by OpenAI
GPT-5.6 Sol (OpenAI, Astra’s predecessor)	65.7%	Reported by OpenAI

All three scores are self-reported by their own labs under what benchlm.ai calls comparable protocol; the tracker notes that different OSWorld 2.0 releases aren't always directly comparable to each other, so treat the gap as directional, not exact.

For Shield, none of that benchmark argument changes what’s in front of its technicians. If Astra reliably logs into old portals and finishes tickets faster than the model it replaced, the OSWorld ranking against Kimi doesn’t show up in the support queue. Benchmark rank matters more to a company shopping for a model than to one already running production workflows on it.

What’s still unverified

Shield’s “thousands of runs a month” has no base number attached, so there’s no way to tell from the case study how much of Forge’s total ticket volume that covers. The customer-satisfaction increase is mentioned without a figure. And computer-use agents working unsupervised in IT-support portals, even “supported” ones with human sign-off, is exactly the kind of deployment OpenAI’s own safety researchers have flagged as risky when access controls are thin; I’d want to see what Shield’s review step actually catches before calling this problem solved.

AI Scoop has covered GPT-6 Astra’s benchmark record before, including a case where a headline overstated what a benchmark actually said. The pattern holds here too: the number in the launch post and the number on an independent tracker aren’t the same number, and both are worth reading before deciding what “best” means for your own use case.

Sources

Note on sources: the Shield numbers come from OpenAI’s own customer-story page, not an independent audit. The OSWorld comparison pairs OpenAI’s launch-post figures for Astra and Sol with Moonshot AI’s self-reported Kimi K3.1 Pro score on benchlm.ai’s public tracker.