What Is Google-Extended and How Does It Differ From Googlebot?

Googlebot controls Search. Google-Extended controls Gemini training and grounding. Here's what each robots.txt token does, what blocking it changes, and what it leaves alone.

Ömer S. Ekinci
CEO, Oxford Impulse
Published
5 min read

You’ve seen the advice to “block AI crawlers”, and now you’re staring at your robots.txt file wondering which line does what. Block the wrong one and you could drop out of Google Search. Block the right one and you’ve opted out of something quite different. So here’s the plain version of Google-Extended against Googlebot.

The short answer

Googlebot is the token for Google Search. Its rules affect Search, including Discover and all Search features. Google-Extended is a separate token that manages whether your crawled content is used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google says it doesn’t affect inclusion in Search and isn’t a ranking signal[1].

What Googlebot controls

Googlebot is Google’s main crawler for Search. Its robots.txt token is Googlebot, and any rule you write for it applies to Google Search as a whole, including Discover and every Search feature[1].

That’s worth pausing on. Disallowing Googlebot is not an “AI-only” setting. It’s a Search setting. If you tell Googlebot to stay out of a folder, you’re telling Search to stay out of that folder. Nobody should use it as a way to opt out of AI training.

What Google-Extended controls

Google-Extended is a standalone token. It has no separate user agent string of its own[1]. Crawling still happens with Google’s existing user agents, and the token is simply a switch you can flip in robots.txt to say how Google may use what it has crawled.

According to Google’s crawler documentation, that switch covers two things[1]:

  • Training future Gemini models.
  • Grounding in Gemini Apps and in Grounding with Google Search on Vertex AI. Grounding means connecting a model’s answer to live sources rather than relying only on what it learned in training.

Google also says Google-Extended doesn’t affect your inclusion in Google Search and isn’t a ranking signal[1]. In other words, Google doesn’t tie it to your place in the results.

The wording matters here: the documentation talks about future models. Don’t read it as a promise that a rule reaches back and undoes anything. It governs how content is used going forward, as the page describes it.

Name the crawler in its own group and disallow everything. Google’s guidance for blocking one crawler is to give it a dedicated group with Disallow: / and leave the others allowed[4]. In practice your file would contain a group that starts with the line User-agent: Google-Extended, followed by the line Disallow: /.

Nothing else needs to change. Googlebot has no rule telling it to stay away, so Search crawling carries on as before.

Want to restrict only part of your site? Use specific paths in place of the slash. The same page of useful rules shows the pattern for blocking a directory rather than a whole site[4].

How Google reads your robots.txt

A few mechanics decide whether your file does what you think it does. These come from Google’s robots.txt specification[3] and its crawler documentation[1].

  • Common crawlers always obey robots.txt. That’s Google’s stated position for its common crawlers[1].
  • One matching token is enough. If a crawler matches more than one of its tokens, a rule for any of them can apply[1].
  • The most specific matching group wins. If you have a group for Googlebot and a general group for all crawlers, Googlebot follows the Googlebot one[3].
  • User-agent values aren’t case-sensitive. Google-Extended and google-extended match the same group[3].
  • Google caches robots.txt for up to 24 hours. Don’t expect a change to take effect the minute you save[3].
  • There’s a size limit of 500 KiB. Content beyond that is ignored, so keep the file lean[3].

Tip

After you edit the file, wait out the cache window before deciding something didn’t work. A 24-hour delay is normal, not a fault.

What neither token controls

This is where most confusion starts, so be exact about it.

Google-Extended doesn’t govern Search. It has no effect on inclusion or ranking[1]. And Google’s documentation ties it to Gemini training and grounding. It doesn’t say the token governs AI Overviews or Google AI Mode, so don’t treat it as a control for those. Our guide to showing up in Google AI Mode is at https://oxfordimpulse.com/how-to-show-up-in-google-ai-mode/ . Google’s own page on AI features and your website is the official reference[2].

The Search Console control isn’t a training control. Google Search Console has a generative AI control, and Google’s help page is explicit that it doesn’t affect AI training. Google-Extended is the token for that[5]. Two switches, two different jobs.

Googlebot isn’t an AI-only lever. We said it above, but it’s the mistake with the highest price, so it earns a second mention. Blocking it affects Search.

A future signal: AIPREF

Google has said it intends to support the AIPREF training signal in early 2027[1]. That’s an intention, not a live feature. Nothing about it changes how you should set up robots.txt today, so plan around Google-Extended for now and check the crawler documentation when the date gets closer.

Should you block Google-Extended?

That’s a business call, and Google’s documentation doesn’t make it for you. What it does give you is a clean trade: Google-Extended doesn’t touch Search inclusion or ranking[1], so the decision isn’t about your organic traffic. It’s about whether you’re comfortable with your content being used to train future Gemini models and for grounding in the products listed above.

Decide per site section if you need to. A public help centre and a paid-content area might deserve different answers, and path-level rules let you do that[4].

FAQs

What is Google-Extended?

It’s a robots.txt token that lets you manage whether Google uses your crawled content to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI[1]. It isn’t a separate crawler with its own user agent string.

Does blocking Google-Extended hurt my Google Search rankings?

Google says it doesn’t. The token doesn’t affect inclusion in Google Search and isn’t a ranking signal[1]. Your Search crawling is handled by Googlebot, which stays untouched.

What happens if I block Googlebot?

You affect Google Search, including Discover and all Search features[1]. It’s not a way to opt out of AI training, and it can cost you visibility, so don’t use it for that purpose.

Does Google-Extended have its own user agent?

No. It’s a standalone token with no separate user agent string, and crawling uses Google’s existing user agents[1]. You use the token only in your robots.txt rules.

How do I block only Google-Extended?

Create a group for Google-Extended and add Disallow: / to it, leaving other crawlers allowed[4]. Googlebot and everything else then carry on under their own rules.

Does the Search Console generative AI control stop AI training?

No. Google’s help page says that control doesn’t affect AI training, and that Google-Extended is the token to use for it[5].

How long do robots.txt changes take to apply?

Google caches robots.txt for up to 24 hours[3]. Allow that long before you check whether a change has taken effect. Keep the file under the 500 KiB limit as well[3].

Does Google-Extended control AI Overviews or AI Mode?

Google’s documentation doesn’t say it does. It describes the token as covering Gemini training and grounding in the products named above[1]. For how Google describes AI features in Search, read its AI features page[2].

Where to go from here

Open your robots.txt today, check whether Google-Extended is named anywhere, and decide on purpose rather than by default. Then read the rest of our work on AI visibility at https://oxfordimpulse.com/ai-visibility/ to see how crawler choices fit with the wider picture of being found in AI answers.

Sources

Share this article

Written by

Ömer S. Ekinci

CEO, Oxford Impulse Google Partner

Ömer S. Ekinci is the director of Oxford Impulse, a London-based Google Partner agency. He works hands-on in Google Ads, Meta Ads and Google Merchant Center accounts for e-commerce, healthcare and B2B brands across the UK, Europe and the US.

Free audit

Want a second pair of eyes?

Get a free audit and a clear list of what to fix first.

Get a free audit
0% read 5 min left

What Is Google-Extended and How Does It Differ From Googlebot?

Get a free audit