AI Training Opt Out in robots.txt: Who Honors It, When

On October 30, OpenAI, Google and Microsoft each said they will honor the same robots.txt entry as a “don’t train on my site” signal. The entry is two lines long. The catch is in the dates: Microsoft says it starts reading it on February 1, 2027, Google says 2027 and OpenAI says “the coming months.” Today, as far as the three announcements say, nobody reads it.

I went through each company’s own post and the draft they’re all pointing at. The announcements share a headline, but the details differ from company to company.

What the line looks like

The signal comes from an Internet-Draft written by Kevin Kelley, who lists Anthropic as his employer. The format is the robots.txt one sites already use, but instead of naming a crawler on the User-Agent line, you name a purpose:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Per the draft, that says “no AI training” for the whole site, except the press-kit folder, where training is fine. Nothing new is added to robots.txt itself, so older parsers should just skip the group as an unknown crawler.

Two rules in the draft matter for anyone planning to use it. A blanket User-Agent: * group does not count as a training preference, and if the file has no __usage_ai-training group, the draft says no preference has been expressed. The draft also says the token is “not a directive for crawling behavior,” so blocking training this way doesn’t keep a crawler off your pages.

Three companies, three timelines

Company	What it said on October 30	When
OpenAI	Will honor the common signal; existing crawler controls still apply	“We aim to implement the signal in the coming months”
Google	Plans to read the token as a Google-Extended disallow if the site hasn’t set Google-Extended itself	“We aim to support this new control along with other AI providers in 2027”
Microsoft	Will honor AI Training preferences in robots.txt, plus new page-level settings	Starting February 1, 2027

Anthropic is the odd one out. Its employee wrote the draft, The Verge lists it among the companies that say they will adopt the standard, and Engadget reports it has already started supporting the directive. I couldn’t find an Anthropic announcement of its own, so treat that as the outlets’ reporting.

OpenAI calls its move a “voluntary effort” and asks other AI providers to honor the signal too. Nothing in the three posts says anyone is obliged to.

Microsoft is changing more than the training signal

Microsoft’s Bing blog goes further than the other two. Microsoft adds a separate page-level setting for “AI grounding,” meaning the use of a page’s content to inform generated answers in Bing, Copilot and Microsoft 365. The options are Allow and Disallow, plus a new one called DisallowExtended, which keeps content out of third-party grounding services but leaves it available in Microsoft’s own products.

Two older tags are retired as well. After March 1, 2027, NoArchive and NoCache stop counting as AI training or grounding preferences. Sites that use them need to publish the new settings before that date. For grounding, Microsoft says to record a temporary preference in Bing Webmaster Tools too, and that option opens on December 15, 2026.

Microsoft also warns that choosing Disallow or DisallowExtended for grounding “will significantly reduce your content’s visibility in AI-generated experiences,” without affecting eligibility for Bing search results.

Google’s post is narrower. It says sites that specifically allow Google-Extended are eligible to appear in Gemini Apps responses through grounding. There’s no separate grounding switch in the announcement.

What it can’t do

The draft’s own front page says it is “not endorsed by the IETF and has no formal standing in the IETF standards process.” It’s an individual submission that expires on May 3, 2027 unless someone updates it. Google’s post frames it as groundwork for “a future standard,” which is a hope, not a fact yet.

Engadget points out another limit: the signal can’t undo past scraping. If a company already crawled your pages, it still has them. Only data collected after a company starts reading the token is affected. And the draft says private agreements “MAY override” preferences expressed this way, so a publisher with a licensing deal could end up with different terms.

None of this adds payment or licensing for publishers. It’s an opt-out switch and nothing more.

My read

This is a real convenience and a small one. Right now a publisher who wants out of training has to find each company’s crawler name and keep the list current. One line beats a pile of per-company rules. But a promise to honor a signal in 2027 is a long way from honoring it, and the gap between “this month” and February is a lot of crawling.

The part I’d watch is grounding. The shared token covers training only, and Microsoft is the only one of the three that put a matching control for answer-generation on the table, with a stated warning that using it costs visibility. That’s the trade publishers actually argue about, and it isn’t solved here. I covered a related fight over scraping terms in Meta’s business terms change, and the question there was the same: who gets to say what a crawler may do with a page. For how Google’s AI answers have fared in court, see the AI Overviews lawsuit ruling.

One more disclosure, since it’s relevant. This site’s own robots.txt explicitly allows GPTBot, ClaudeBot and a long list of others, so the signal changes nothing for us. Sites that want out should wait for each company’s implementation notes, and check each vendor’s documentation, because the exact tokens and menu names can change before they ship.

Note on sources

The dates and quotes for OpenAI, Google and the IETF draft come from their own pages, read in full. Microsoft’s dates come from the Bing blog excerpts and Search Engine Land’s summary of it, because I read only parts of the Bing post. The claim that Anthropic has started supporting the signal comes from Engadget and The Verge, not from Anthropic.

OpenAI, Google and Microsoft will read one robots.txt line to skip AI training. None of them reads it yet.