Anthropic gave 50 analysts an AI whose opinions were set by coin flips. They agreed with it 86% of the time.
The trial used Claude Code and a made-up farming co-op. Anthropic says the effect was deliberately strengthened, but the part about credit is harder to shrug off.
The number is 86%. Anthropic published a study on October 27 in which 50 professional analysts did a statistical task with Claude Code, and on four judgment calls they ended up on the same side as their agent 86% of the time. Anyone who ignored the agent would land near 50%.
What makes it more than a survey is the setup. Which side the agent took was decided by a coin flip, and the analysts weren’t told.
How the trial worked
Anthropic’s alignment team (Amaç Herdağdelen, Hailey Joren and Alex Tamkin) recruited 61 people who had passed a causal-inference screening test and used Python or R regularly. Eleven chose to work without the agent and were left out of most of the analysis, which leaves 50. Each worked in a browser-based VS Code with Claude Code running Claude Opus 4.8 at high effort. The median session took 65 minutes.
The task was set at a fictional farming co-op. The board would expand a paid advisory program only if it raised farm productivity by roughly 3 to 4%, and it didn’t say how productivity should be measured. Four choices were left open, each with two defensible options: yield per hectare or per unit of cost, one comparison method or another, the enrolled farms or the whole membership, and equal weights or weights by farm size. Anthropic says each choice could move the estimate by several percentage points.
A hidden system prompt gave the agent a lean on each choice, set by coin flip. It was told to argue for that lean as its own view, to change its mind if the evidence turned against it, and never to mislead.
| Decision | Analysts who matched the agent’s lean |
|---|---|
| Outcome measure | 92% |
| Comparison method | 82% |
| Which farms are covered | 81% |
| Weighting | 90% |
| All four combined | 86% |
What the 86% hides
Some analysts had picked an option before the agent spoke. When the agent disagreed with them they switched 60% of the time, against 6% when it agreed. But that comes from only 38 cases, and Anthropic’s own confidence interval runs from 24 to 82 percentage points, so treat 60% as a rough direction.
The final recommendation moved less: 54% recommended rolling out the program when the agent leaned that way, 36% when it leaned the other way. Anthropic’s explanation is that once an analyst has an estimate, the recommendation largely followed from whether the estimate cleared the board’s 3 to 4% threshold. By then the four earlier calls had already shaped the estimate, though. A model fed only the four coin flips predicted 82% of the final recommendations, against 61% for always guessing the most common answer (analysts who proposed a one-year extension are excluded from that comparison).
Then there’s credit. Asked afterward why they chose as they did, 77% of analysts repeated the agent’s argument in at least one answer, sometimes word for word, and only 15% said the agent was where it came from. They couldn’t look back at their chats while answering. Those who followed the agent on all four choices rated their confidence 71 out of 100 on average, the others 73. And 52% called the final recommendation more their own than the agent’s. Six of 48 guessed the study was about the AI’s influence. None suspected the agent’s preferences had been manipulated.
Anthropic’s own caveats
The company is blunt that this may not carry over. It strengthened the agents’ preferences on purpose, so the rates describe an agent that leans and says so. With no lean assigned, Claude Opus 4.8 made the same five choices in 112 of 123 runs, which suggests its default preferences are weaker. It was one task in one field with one model, the analysts knew little about farming, and real analysts would have stronger views and could interview the board. Anthropic also didn’t record how strongly people preferred an option, so some of the deference may just be people with no opinion yet.
It adds that this isn’t a case against using AI. The analysts with the agent finished faster (65 minutes median against 90) and six of the 11 who worked alone made serious methodological errors, against three of 50 with the agent. Anthropic warns the groups aren’t comparable, since the solo group was smaller and less experienced.
My read
The credit finding bothers me more than the 86%. Deferring to a confident colleague isn’t new. Writing the colleague’s argument into your own report as your own reasoning, and feeling just as sure afterward, is what makes a review hard to trust.
I also think the weakest piece of the evidence is the lean itself. Real agents aren’t handed a coin flip and told to argue it. They have default tilts that nobody has mapped well, and this study can’t say how big those are in everyday work. Anthropic itself says the effect may be smaller when agents aren’t told to voice a preference, so I’d read 86% as what a pushy agent can do, not a typical rate.
Still, Anthropic is a company that sells this product, and it chose to publish a result that makes “a human checked it” sound weaker. That counts for something. Its safety-research angle is the sharpest one: if researchers use agents to run and analyze experiments, a human reviewer is only an independent check when the judgment is theirs. A related worry about agents doing things nobody checked runs through Anthropic’s earlier report on agents behaving in ways nobody intended.
Anthropic lists three areas for future work: ways to measure deferral in ordinary work, agents that make the case for each option or ask for the expert’s view first, and institutions that decide which sign-offs need a person’s own informed judgment. It offers no finished fix for any of them.
Note on sources
I read Anthropic’s research page in full. I did not open the linked PDF, so details from the full paper that the page doesn’t show are not included. Al Jazeera and Fox News both repeat the headline figures, going by the excerpts I read. The study is Anthropic describing its own experiment, and I found no outside analysis of it.