You want your pages to show up when someone asks ChatGPT a question, but you don’t want your content feeding the next model. Then you open your robots.txt file and find several OpenAI names that all look alike. Which ones do you allow, and which do you block?
Allow OAI-SearchBot if you want to appear in ChatGPT search, and disallow GPTBot if you don’t want your content used for training. They’re separate settings, so one doesn’t force the other[1]. ChatGPT-User is a different case, covered below.
What each OpenAI crawler does
OpenAI documents its crawlers on a single page[1]. It’s short, and that’s useful, because there isn’t much to untangle. Each bot has one job, and OpenAI treats the controls as independent.
OAI-SearchBot: the one for ChatGPT search
OAI-SearchBot is used for search. OpenAI recommends allowing it if you want your site to be eligible to appear in ChatGPT search results[1]. If you opt out, your pages won’t be shown in ChatGPT search answers. They can still appear as navigational links[1].
That matches how OpenAI describes the product. ChatGPT search gives answers with inline citations and a Sources list, and it draws on third-party search providers and partner content[2]. OpenAI has also said any website or publisher can choose to appear in it[2][3]. So the crawler is one route into search, not the only one.
GPTBot: the one for training
GPTBot crawls content that may be used to train generative AI foundation models. Disallowing it signals that your content shouldn’t be used for that purpose[1]. Publishers can appear in search even if they opt out of training[2][3]. That’s the point of keeping the controls apart.
Watch out
A disallow rule is a signal, not a removal tool. OpenAI’s page doesn’t say blocking GPTBot removes content that was already used, so don’t promise that to anyone, including your legal team.
ChatGPT-User: the odd one out
ChatGPT-User handles certain user actions in ChatGPT and Custom GPTs. Think of it as a visit triggered by a person’s request rather than a crawl[1]. Two things to know. OpenAI says robots.txt rules may not apply to it. And it doesn’t decide whether your site is included in search[1].
So blocking or allowing ChatGPT-User won’t change your search eligibility. Don’t count on a robots.txt line to control it either, because the page says only that the rules may not apply. It doesn’t say what happens in every case.
OAI-AdsBot: for ads only
OAI-AdsBot validates pages that are submitted as ads. OpenAI says its data isn’t used to train foundation models[1].
A robots.txt setup for GPTBot and OAI-SearchBot
If you want search but not training, this is the pattern. Each bot gets its own group in the file:
User-agent: OAI-SearchBot
Allow: /User-agent: GPTBot
Disallow: /
Upload it to the root of your domain, so it lives at yourdomain.com/robots.txt. If you already have rules for other crawlers, add these groups without disturbing them. If you only want to block training for part of the site, use specific paths in the GPTBot group instead of a single slash.
Good to know
OpenAI says a robots.txt change can take about 24 hours to affect search[1]. Don’t edit the file, test ChatGPT ten minutes later and decide it failed.
What happens if you allow both
This one catches people out. If you allow both the search bot and the training bot, OpenAI may use the results of one crawl for both purposes[1]. That’s a reason to be deliberate. If you want search without training, disallow GPTBot explicitly rather than leaving everything open and hoping the uses stay separate.
Checking that the traffic is really OpenAI
User-agent strings can be faked. OpenAI publishes IP ranges for each bot as JSON files, linked from its crawler page[1]. If your server logs show a hit claiming to be OAI-SearchBot, compare the IP address against the published list before you treat it as genuine. The same goes if you’re thinking about firewall rules.
What OpenAI’s page doesn’t say
The documentation tells you what each bot is for and how to control it. It doesn’t say how pages are selected for search answers, and it doesn’t promise rankings, citations or traffic. Nothing in it says you need special markup or tracking parameters. Be careful with confident claims, from anyone, about how to get cited.
FAQs
Which OpenAI crawlers should I allow in robots.txt?
Allow OAI-SearchBot if you want your site to be eligible for ChatGPT search. OpenAI recommends this[1]. Decide on GPTBot separately, since it relates to training rather than search.
Can I appear in ChatGPT search and block training?
What does GPTBot do?
GPTBot crawls content that may be used to train generative AI foundation models. Disallowing it signals that your content shouldn’t be used for training[1].
What happens if I block OAI-SearchBot?
Your site won’t be shown in ChatGPT search answers. It can still appear as a navigational link[1].
Does robots.txt control ChatGPT-User?
Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User, which handles certain user actions in ChatGPT and Custom GPTs[1]. It also doesn’t decide search inclusion.
How long does a robots.txt change take to work?
OpenAI says it can take about 24 hours for a change to affect search[1]. Wait at least that long before you judge the result.
Does blocking GPTBot remove content already used for training?
OpenAI’s crawler page doesn’t say so. It describes the disallow rule as a signal that content shouldn’t be used for training[1].
What is OAI-AdsBot?
It validates pages submitted as ads. OpenAI says its data isn’t used to train foundation models[1].
Where to go from here
Open your robots.txt today, add the two groups above if they suit you, and set a reminder to check ChatGPT search again after 24 hours. If you sell products and want to understand the commercial side of appearing in ChatGPT, read our guide at https://oxfordimpulse.com/sell-your-products-with-chatgpt/ .
Sources
- Overview of OpenAI crawlers developers.openai.com
- Introducing ChatGPT search openai.com
- SearchGPT prototype openai.com
