On 30 October Google said it plans to honour one shared "no AI training" signal in a website's robots.txt file, and a lot of headlines read it as "block AI now". So should you block AI crawlers, or opt out of Google's AI features? Those are two different switches, and one of them costs you traffic.
Short answer: blocking Google's AI training crawler does not remove you from Google Search, and Google says so. Opting out in Search Console is different: Google says those sites get no traffic or impressions from its AI features, like AI Overviews. For a local business that wants to be found, I'd leave both alone and just read what your robots.txt says.
What did Google announce on 30 October?
John Mueller of Google Search posted on the Search Central blog that website owners could soon use one shared "policy token" in robots.txt to set a single AI training preference across participating AI providers. If you opt out of training with it and don't name Google-Extended, Google "plans to interpret that as if the publisher had disallowed" Google-Extended. Existing company controls keep working. Google says, "We aim to support this new control along with other AI providers in 2027."
It is worth being clear about what this is not. It isn't live. The post doesn't publish a token name, so I won't guess one. It doesn't mention AI Overviews or ranking. It is also a different story from the rule on AI-written text that I covered in whether Google penalizes AI content. That one was about what you publish. This one is about who may use it.
What does each Google control actually do?
There are three, and owners mix them up. Here they are in Google's own words, as I read them on 3 November 2026.
| Control | Where it lives | What Google says it does | Status |
|---|---|---|---|
| Google-Extended | A token in robots.txt | Lets publishers manage whether content may be used to train future Gemini models. It "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." | Live, offered since 2023 |
| Search generative AI control | Search Console, whole site | Manages your site's inclusion in generative AI features on Google Search, including AI Overviews and AI Mode. "Sites that opt out will not receive traffic or impressions from our generative AI features." | Live (announced 3 June 2026) |
| Shared AI training control | A shared policy token in robots.txt | One training preference across participating AI providers. Google plans to treat it as disallowing Google-Extended. | A proposal, "in 2027". No token name published |
The correction first: blocking Google-Extended doesn't touch Google Search. Google says it has no separate user agent string, only a token, and I won't explain how to write one. The Search switch is a different kind of thing. It sits in Search Console, it is about AI Overviews and AI Mode, and Google's blog says it isn't a ranking signal outside those features. So neither control removes you from normal results.
Neither one is free, though. The 30 October post says "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps." So allowing it can earn you a mention in Gemini. In my view, "lose" is the right word for the Search switch, and for most local businesses I wouldn't pay it.
Is a robots.txt line a lock?
No. It's a request. The standard behind robots.txt, RFC 9309, says it plainly: "These rules are not a form of access authorization." Google says its own common crawlers "always obey robots.txt rules", but that is Google describing Google. I can't tell you what every other company's crawler does, and I didn't test any. I tried to read OpenAI's crawler page on 3 November and it wouldn't open for me (a 403 error), so I won't say what its crawlers do.
A line only covers the crawlers it names. The shared control is a plan for 2027. And Google doesn't say what happens to content it already used for Gemini training, so I won't guess. If you hold content that is valuable because nobody else has it, treat robots.txt as a polite sign on the door, not a lock. For how AI tools pick which sites to mention at all, see how to get recommended by AI.
Where do I fit, and what would I check?
I read my own file at eaglerockwebdesign.com/robots.txt on 3 November 2026. It has one group for all crawlers, plus 14 named ones: Googlebot, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, anthropic-ai, PerplexityBot, Google-Extended, Applebot, Applebot-Extended, CCBot and meta-externalagent. Every one is allowed. I did that on purpose, because I want ChatGPT, Perplexity or Gemini to be able to read my site and mention it. That's my choice for my site, not advice for everyone. Since Google-Extended is allowed, my site sits in the group Google describes as eligible for Gemini grounding.
Here's how I'd think about yours. If you run a clinic, a trade or an accounting firm that gets calls, do nothing, read your file, and keep the Search AI control on. If your main asset is original photos, paid writing or members-only content, blocking training is a reasonable call and it's yours to make. If you're unsure, don't switch anything off this month. Opting out of the Search AI features is a business decision, not a security measure.
On my sites, this kind of thing is set up when I build them. You can see how I build search and AI search into the sites I make and what a custom-coded website involves. If your file already looks fine, you don't need me. If you'd like a second pair of eyes, I'll look at your site for free with a free website review. I only take on 4 new sites a month, because I build every site myself.
What to do this week:
- Type /robots.txt after your domain and read what it says. No editing needed.
- Ask whoever built your site, or your host, whether it was set on purpose and what it says for Google-Extended.
- Open Search Console and check whether anyone turned the Search generative AI control off.
Related posts
- How to Get Your Business Recommended by AI
- Does Google Penalize AI Content?
- Google Business Profile Checklist for Toronto Small Businesses
- Best Website Builders 2026
Sources
- Google Search Central Blog: Supporting a shared AI training control for websites (John Mueller, 30 October 2026, read 3 November 2026)
- Google Search Central: Google's common crawlers (no date shown, read 3 November 2026)
- Google Search Central: Control what you share with Google Search (read 3 November 2026)
- Google Search Central: Documentation updates (21 October 2026 entry, read 3 November 2026)
- Google, The Keyword: New opportunities, control and insights for website owners (3 June 2026, page shows updated 31 August 2026, read 3 November 2026)
- IETF: RFC 9309, Robots Exclusion Protocol (September 2022)
Questions people ask
Does blocking AI crawlers hurt my Google ranking?
Google says Google-Extended does not impact inclusion in Google Search and is not a ranking signal, and that the Search generative AI control is not a ranking signal outside its AI features. I have not measured it on any site.
How do I see what my robots.txt says?
Type /robots.txt after your domain in a browser. It's a public file. Read it, and ask whoever built your site what the lines for each crawler are meant to do.
Does ChatGPT or Perplexity use Google-Extended?
Google-Extended is Google's token, set by Google for Google. Other companies have their own crawlers and their own pages. I couldn't open OpenAI's on 3 November 2026, so I don't say what they do.
Is Google-Extended the same as Googlebot?
No. Google's page says Google-Extended is a standalone product token with no separate user agent string, and that it doesn't affect inclusion in Google Search. Googlebot is the crawler that does Search.
Not sure what your site tells Google and AI tools?
I'll look at it for free. Ask for a free website review first, or just get in touch.
Get a Free Quote