Checked November 8, 2026 · Labels: Cloud (how AI providers use your site)
Short answer: today you opt out of AI training one company at a time, with tokens such as GPTBot (OpenAI) and Google-Extended (Google) in your robots.txt. A new shared token, __usage_ai-training, is planned so one line can speak to several AI providers at once, but it is not live yet: Google says it aims to support it in early 2027, and OpenAI says "in the coming months." If you also want to be cited in Gemini, there is a catch worth knowing before you copy anything: Google says sites that specifically allow Google-Extended stay eligible for Gemini grounding clicks, and a broad training opt-out may otherwise count as a "no" to Google-Extended. With Google, that same switch also covers Gemini training, so for now you can't allow citations while opting out of training.
This guide follows one reader: someone who runs a small recipe-and-gardening blog, would rather their posts did not end up training AI models, but still wants AI assistants to cite and link to them.
What changed on October 30
On October 30, 2026, Google and OpenAI both announced plans to support a shared "policy token" in robots.txt for AI training preferences, __usage_ai-training, drawn from an Internet Engineering Task Force (IETF) draft.
What the documentation says, in plain terms:
- It is planned, not live. Google "aims to support this new control along with other AI providers in 2027," and its crawler documentation says early 2027. OpenAI says it aims to implement the signal "in the coming months." The IETF document is a draft, which by its own terms is "work in progress."
- Google will read it as Google-Extended, unless you say otherwise. If your robots.txt uses the shared token to opt out of training and does not explicitly mention
Google-Extended, Google "plans to interpret that as if the publisher had disallowed Google-Extended." - Allowing Google-Extended keeps Gemini citations possible. Google says sites that "specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps."
- The old tokens keep working. Google and OpenAI both say
Google-Extended,GPTBotandOAI-SearchBotcontinue alongside the shared one. User-agent: *does not count. The draft says a wildcard group must not be read as the training preference. If there is no__usage_ai-trainingline, no preference has been expressed.
Training, citations and search are separate switches
What each token covers, according to each company's own documentation:
| Token | Whose | What it controls | In effect today (per docs)? |
|---|---|---|---|
Google-Extended | Whether Google may use your content to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google says it does not affect inclusion or ranking in Google Search | Yes | |
GPTBot | OpenAI | Crawling for content that may be used to train OpenAI's generative models | Yes |
OAI-SearchBot | OpenAI | Whether your site can appear in ChatGPT search answers | Yes |
__usage_ai-training | Shared (IETF draft) | A training preference for participating AI providers; not a crawl rule | Not yet (planned) |
Googlebot | Google Search crawling. Leave this alone if you want search traffic | Yes |
From Google's crawler documentation and Search Central blog, OpenAI's crawler overview and announcement, and the IETF draft, checked November 8, 2026.
The catch for our blogger: Google-Extended is one switch for two things. Per Google's documentation, allowing it to keep Gemini grounding also means allowing it for Gemini training. Neither Google nor the draft offers a way, as of today, to say "cite me, but don't train on me" to Google. You have to choose which matters more.
Three choices, three robots.txt files

Each snippet goes into the robots.txt file at the root of your site (for example yourblog.com/robots.txt). Keep your existing rules for Googlebot and other search crawlers as they are.
Choice A: block training broadly (once the shared token is live)
# Shared AI training preference (planned; Google support aimed for early 2027)
User-agent: __usage_ai-training
Disallow: /
Based on the announcements, participating providers should read this as "no training" for every page. Because Google-Extended is not mentioned, Google plans to treat it as disallowing Google-Extended, so your site would likely not be eligible for Gemini Apps grounding clicks. The draft says existing parsers ignore groups they don't recognize, so adding it early should be harmless, but don't rely on it alone yet.
Choice B: block training broadly, keep Gemini grounding
# Shared AI training preference (planned)
User-agent: __usage_ai-training
Disallow: /
# Explicitly allow Google-Extended to stay eligible for Gemini grounding
User-agent: Google-Extended
Allow: /
This is the setup for our blogger. Google's announcement says the shared token is read as a Google-Extended "no" only when Google-Extended is not explicitly specified, and that sites specifically allowing it stay eligible for Gemini grounding clicks. The trade-off, again: allowing Google-Extended also allows Gemini training.
Choice C: block OpenAI training today
# Opt out of OpenAI model training
User-agent: GPTBot
Disallow: /
# Stay eligible for ChatGPT search answers
User-agent: OAI-SearchBot
Allow: /
This works now, according to OpenAI's crawler documentation: disallowing GPTBot says your content should not be used to train its generative models, while allowing OAI-SearchBot keeps your pages eligible for ChatGPT search (OpenAI says changes take about 24 hours). You can add Choice A or B later; OpenAI says its existing controls will keep working alongside the shared signal.
If you want the Google equivalent today, Google's crawler documentation says a User-agent: Google-Extended group with Disallow: / is honored now, with the same grounding trade-off as Choice A.
Choose A if…
- Keeping your writing out of AI training matters more to you than Gemini citations.
- You are happy to add a line now and wait for providers to switch it on.
Choose B if…
- You want fewer providers training on your work, but Gemini answers linking to your recipes bring you readers.
- You accept that, with Google specifically, "cite me" and "train on me" are currently the same switch.
Choose C if…
- You want something that works today, and to stay visible in ChatGPT search while opting out of OpenAI training.
Three things robots.txt will not do
- It is a preference, not a lock. Robots.txt is honored voluntarily, and the draft itself says private agreements may override these preferences.
- It does not undo the past. These signals are about future use. They don't remove content already collected.
- It does not cover what you paste into chatbots. That is about your own uploads: see what happens before you upload a document, our private AI options guide for drafting without a cloud provider, and what local apps still send online.
More for local businesses: AI tools for local businesses, our map of every guide for shops, salons, cafes and trades.
Frequently asked questions
Does blocking Google-Extended hurt my Google Search rankings?
Google says no. Its crawler documentation states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."
Is __usage_ai-training working yet?
Not as of November 8, 2026. Google aims to support it in early 2027, and OpenAI says in the coming months. It is still an IETF draft.
Will User-agent: * with Disallow: / block AI training?
It blocks every crawler that obeys it, including search, which is rarely what a blog wants. Under the draft it does not count as a training preference either. Use the named tokens.
How we researched this: a documentation-based guide, not a crawler test. We did not deploy these files or monitor bot traffic; we report what each company and the draft say. Sources checked November 8, 2026: Google Search Central blog, "Supporting a shared AI training control for websites" (October 30, 2026, https://developers.google.com/search/blog/2026/10/ai-training-control) · Google's common crawlers documentation (Google-Extended entry, updated October 30, 2026, https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) · OpenAI, "Making it easier for websites to express AI training preferences" (October 30, 2026, https://openai.com/index/helping-websites-express-ai-training-preferences/) · OpenAI crawler overview (https://developers.openai.com/api/docs/bots) · IETF Internet-Draft draft-kelley-aipref-training-preference-signal-00 (https://datatracker.ietf.org/doc/html/draft-kelley-aipref-training-preference-signal-00). Announcements describe plans that may change; recheck before relying on them. See our editorial policy. Illustrations are AI-generated artwork and do not depict a tested product.
