Skip to content

THE FIELD JOURNAL

How to Block AI Training Crawlers (and the Gemini Citations Trade-off)

By Local AI Scout · Beginner guide

Illustration of a small robot outside a garden fence, with flowers behind it and a sign showing a crossed-out robot

Checked November 8, 2026 · Labels: Cloud (how AI providers use your site)

Short answer: today you opt out of AI training one company at a time, with tokens such as GPTBot (OpenAI) and Google-Extended (Google) in your robots.txt. A new shared token, __usage_ai-training, is planned so one line can speak to several AI providers at once, but it is not live yet: Google says it aims to support it in early 2027, and OpenAI says "in the coming months." If you also want to be cited in Gemini, there is a catch worth knowing before you copy anything: Google says sites that specifically allow Google-Extended stay eligible for Gemini grounding clicks, and a broad training opt-out may otherwise count as a "no" to Google-Extended. With Google, that same switch also covers Gemini training, so for now you can't allow citations while opting out of training.

This guide follows one reader: someone who runs a small recipe-and-gardening blog, would rather their posts did not end up training AI models, but still wants AI assistants to cite and link to them.

What changed on October 30

On October 30, 2026, Google and OpenAI both announced plans to support a shared "policy token" in robots.txt for AI training preferences, __usage_ai-training, drawn from an Internet Engineering Task Force (IETF) draft.

What the documentation says, in plain terms:

  • It is planned, not live. Google "aims to support this new control along with other AI providers in 2027," and its crawler documentation says early 2027. OpenAI says it aims to implement the signal "in the coming months." The IETF document is a draft, which by its own terms is "work in progress."
  • Google will read it as Google-Extended, unless you say otherwise. If your robots.txt uses the shared token to opt out of training and does not explicitly mention Google-Extended, Google "plans to interpret that as if the publisher had disallowed Google-Extended."
  • Allowing Google-Extended keeps Gemini citations possible. Google says sites that "specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps."
  • The old tokens keep working. Google and OpenAI both say Google-Extended, GPTBot and OAI-SearchBot continue alongside the shared one.
  • User-agent: * does not count. The draft says a wildcard group must not be read as the training preference. If there is no __usage_ai-training line, no preference has been expressed.

Training, citations and search are separate switches

What each token covers, according to each company's own documentation:

TokenWhoseWhat it controlsIn effect today (per docs)?
Google-ExtendedGoogleWhether Google may use your content to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google says it does not affect inclusion or ranking in Google SearchYes
GPTBotOpenAICrawling for content that may be used to train OpenAI's generative modelsYes
OAI-SearchBotOpenAIWhether your site can appear in ChatGPT search answersYes
__usage_ai-trainingShared (IETF draft)A training preference for participating AI providers; not a crawl ruleNot yet (planned)
GooglebotGoogleGoogle Search crawling. Leave this alone if you want search trafficYes

From Google's crawler documentation and Search Central blog, OpenAI's crawler overview and announcement, and the IETF draft, checked November 8, 2026.

The catch for our blogger: Google-Extended is one switch for two things. Per Google's documentation, allowing it to keep Gemini grounding also means allowing it for Gemini training. Neither Google nor the draft offers a way, as of today, to say "cite me, but don't train on me" to Google. You have to choose which matters more.

Three choices, three robots.txt files

Illustration of three garden entrances: a closed gate, an entrance with a visitor passing through, and a partly open divided gate
Three choices: close the gate to training, close it but keep a door open for citations, or close one lane today. Illustrative artwork, not a product screenshot.

Each snippet goes into the robots.txt file at the root of your site (for example yourblog.com/robots.txt). Keep your existing rules for Googlebot and other search crawlers as they are.

Choice A: block training broadly (once the shared token is live)

# Shared AI training preference (planned; Google support aimed for early 2027)
User-agent: __usage_ai-training
Disallow: /

Based on the announcements, participating providers should read this as "no training" for every page. Because Google-Extended is not mentioned, Google plans to treat it as disallowing Google-Extended, so your site would likely not be eligible for Gemini Apps grounding clicks. The draft says existing parsers ignore groups they don't recognize, so adding it early should be harmless, but don't rely on it alone yet.

Choice B: block training broadly, keep Gemini grounding

# Shared AI training preference (planned)
User-agent: __usage_ai-training
Disallow: /

# Explicitly allow Google-Extended to stay eligible for Gemini grounding
User-agent: Google-Extended
Allow: /

This is the setup for our blogger. Google's announcement says the shared token is read as a Google-Extended "no" only when Google-Extended is not explicitly specified, and that sites specifically allowing it stay eligible for Gemini grounding clicks. The trade-off, again: allowing Google-Extended also allows Gemini training.

Choice C: block OpenAI training today

# Opt out of OpenAI model training
User-agent: GPTBot
Disallow: /

# Stay eligible for ChatGPT search answers
User-agent: OAI-SearchBot
Allow: /

This works now, according to OpenAI's crawler documentation: disallowing GPTBot says your content should not be used to train its generative models, while allowing OAI-SearchBot keeps your pages eligible for ChatGPT search (OpenAI says changes take about 24 hours). You can add Choice A or B later; OpenAI says its existing controls will keep working alongside the shared signal.

If you want the Google equivalent today, Google's crawler documentation says a User-agent: Google-Extended group with Disallow: / is honored now, with the same grounding trade-off as Choice A.

Choose A if…

  • Keeping your writing out of AI training matters more to you than Gemini citations.
  • You are happy to add a line now and wait for providers to switch it on.

Choose B if…

  • You want fewer providers training on your work, but Gemini answers linking to your recipes bring you readers.
  • You accept that, with Google specifically, "cite me" and "train on me" are currently the same switch.

Choose C if…

  • You want something that works today, and to stay visible in ChatGPT search while opting out of OpenAI training.

Three things robots.txt will not do

  1. It is a preference, not a lock. Robots.txt is honored voluntarily, and the draft itself says private agreements may override these preferences.
  2. It does not undo the past. These signals are about future use. They don't remove content already collected.
  3. It does not cover what you paste into chatbots. That is about your own uploads: see what happens before you upload a document, our private AI options guide for drafting without a cloud provider, and what local apps still send online.

More for local businesses: AI tools for local businesses, our map of every guide for shops, salons, cafes and trades.

Frequently asked questions

Does blocking Google-Extended hurt my Google Search rankings?

Google says no. Its crawler documentation states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."

Is __usage_ai-training working yet?

Not as of November 8, 2026. Google aims to support it in early 2027, and OpenAI says in the coming months. It is still an IETF draft.

Will User-agent: * with Disallow: / block AI training?

It blocks every crawler that obeys it, including search, which is rarely what a blog wants. Under the draft it does not count as a training preference either. Use the named tokens.


How we researched this: a documentation-based guide, not a crawler test. We did not deploy these files or monitor bot traffic; we report what each company and the draft say. Sources checked November 8, 2026: Google Search Central blog, "Supporting a shared AI training control for websites" (October 30, 2026, https://developers.google.com/search/blog/2026/10/ai-training-control) · Google's common crawlers documentation (Google-Extended entry, updated October 30, 2026, https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) · OpenAI, "Making it easier for websites to express AI training preferences" (October 30, 2026, https://openai.com/index/helping-websites-express-ai-training-preferences/) · OpenAI crawler overview (https://developers.openai.com/api/docs/bots) · IETF Internet-Draft draft-kelley-aipref-training-preference-signal-00 (https://datatracker.ietf.org/doc/html/draft-kelley-aipref-training-preference-signal-00). Announcements describe plans that may change; recheck before relying on them. See our editorial policy. Illustrations are AI-generated artwork and do not depict a tested product.

Keep exploring: Apps · Models · All guides

THE SCOUT LETTER

Good finds.
Less noise.

Useful local AI apps, practical guides and the updates worth your time. A thoughtful note in your inbox.

Free to join. Unsubscribe with one click.