GPTBot is OpenAI’s web crawler for collecting content that may be used to train its AI models. It identifies itself with the user agent token “GPTBot” and follows the rules in your robots.txt file. GPTBot is one of several OpenAI agents: OAI-SearchBot crawls pages for ChatGPT search, and ChatGPT-User fetches pages when a person asks ChatGPT to visit them. Blocking GPTBot opts your site out of training without removing it from ChatGPT search, as long as you leave OAI-SearchBot allowed.
This guide explains what this crawler does, how it differs from OpenAI’s other bots, how to allow or block it, how to check whether it visits your site, and how to decide what is right for your business.

What is GPTBot?
OpenAI introduced GPTBot in 2023 and published documentation so site owners could control it. The crawler visits public web pages and collects content that may be used to improve future models. OpenAI says it filters out sources that require paywall access, are known to gather personal information, or violate its policies.
You can spot it in your server logs by its user agent string, which contains “GPTBot” along with a version number and a link to OpenAI’s documentation. OpenAI also publishes the IP address ranges GPTBot uses, so you can confirm that a request claiming to be GPTBot comes from OpenAI.
Why AI crawlers became a business decision
For most of the web’s history, crawler policy was a technical detail. Site owners let search engines in because search sent visitors back. AI crawlers changed that trade. A training crawler takes content to improve a model, and the model may answer questions later without sending anyone to the original page. A search crawler for an AI product may cite and link your page, or it may summarize it so well that the reader never clicks.
That is why media companies, publishers and many software companies now treat AI crawler access as a business question. Some have signed licensing deals with AI companies. Others block training crawlers while they negotiate or wait to see how the market settles. Many marketing sites take the opposite view and open their doors, because being known by AI assistants is part of how buyers find them.
The right policy depends on your model. If people pay for your content, protect it. If your content exists to attract customers, distribution is usually worth more than control.
GPTBot vs OAI-SearchBot vs ChatGPT-User
OpenAI documents separate agents for different jobs. Mixing them up is the most common mistake site owners make.
| Agent | Purpose | Controlled by robots.txt? | Effect of blocking |
|---|---|---|---|
| GPTBot | Collects content that may be used for model training | Yes | Your content is not used for training from that point on |
| OAI-SearchBot | Crawls pages so they can appear in ChatGPT search results | Yes | Your pages are not shown in ChatGPT search answers |
| ChatGPT-User | Fetches a page when a user asks ChatGPT or a custom GPT to visit it | OpenAI says robots.txt rules may not apply, since a person starts the request | Unreliable through robots.txt alone |
OpenAI has also listed an agent for validating ads. The list changes over time, so check OpenAI’s crawler documentation before you edit your rules.
The practical takeaway: GPTBot controls training, and OAI-SearchBot controls whether you can appear in ChatGPT search. They are independent, so you can block one and allow the other.
How to block GPTBot
To stop it from crawling your whole site, add this to the robots.txt file at the root of your domain:
User-agent: GPTBot
Disallow: /
To block it from some folders but not others:
User-agent: GPTBot
Allow: /blog/
Disallow: /members/
Disallow: /internal-docs/
Each crawler reads only the group of rules that names it (or the wildcard group if none does), so a rule for GPTBot does not affect OAI-SearchBot or Googlebot.
How to allow GPTBot
If your robots.txt has no rule naming OpenAI’s training crawler and no wildcard rule that blocks it, the crawler can already visit your site. To make it explicit:
User-agent: GPTBot
Allow: /
Explicit allow rules are useful when your robots.txt has a broad wildcard disallow for other bots, since they make your intent clear.
Block training, stay in ChatGPT search
Many publishers want this combination. They do not want their content used for training, but they do want to appear when people search in ChatGPT. Use:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
OpenAI notes that changes to search-related rules can take around a day to take effect. Our guide to ChatGPT SEO covers what else helps pages appear in ChatGPT answers.

Should you block GPTBot?
There is no single right answer. It depends on what your content is worth and how you make money from it.
Reasons to block the training crawler
- Your content is your product, such as paid research, news, courses or proprietary data.
- You plan to license content to AI companies and want to keep that option open.
- Your legal team prefers not to contribute content to model training.
- Crawl traffic is costing you server resources.
Reasons to allow the training crawler
- You want future models to know your brand, products and point of view.
- Your content is marketing, and wider distribution helps you.
- You publish documentation that you want AI assistants to explain accurately.
- Blocking training does little for your goals compared with the visibility you might give up over time.
Most marketing sites allow both OpenAI crawlers because they want brand awareness everywhere. Many news publishers block the training crawler and decide separately on search. Whatever you choose, make it a deliberate decision rather than a default.
How AI crawlers differ from search engine crawlers
Search crawlers like Googlebot and Bingbot index pages so they can rank them and send visitors. That exchange has been understood for decades: you let the crawler in, and in return you can earn traffic. AI training crawlers collect pages to teach a model about language and the world. They do not rank your page or send visitors directly. AI search crawlers sit in between. They index pages so an assistant can cite them in answers, which can send some visitors, though often fewer than a classic search result.
Because these jobs are different, it makes sense to treat each type of crawler on its own terms. A blanket “block all AI” rule throws away the search visibility you might want, while a blanket “allow all” rule gives away training access you might want to keep. Writing a separate rule for each agent takes a few minutes and gives you control over each trade.
Keep the rules readable. Group AI agents together in robots.txt with a short comment line explaining your policy, so the next person who edits the file understands why each rule is there.
A simple decision framework
| Your situation | Training crawler | Search crawler |
|---|---|---|
| Marketing site, SaaS, local business | Allow | Allow |
| Documentation or help center | Allow | Allow |
| Ad-supported publisher | Often block | Usually allow |
| Paid research, courses or data | Block | Allow public summary pages only |
| Private or member content | Block and require login | Block and require login |
Revisit the decision once a year, or whenever your business model changes. The list of crawlers and their rules changes quickly, and a choice that made sense two years ago may not fit your goals today.
Server load and crawl budget
Some site owners first notice AI crawlers because of server load. Busy crawlers can request thousands of pages in a short time, which can slow small sites on shared hosting. If that happens, you have options short of a full block. You can disallow heavy sections that add little value, such as search result pages, tag archives and faceted filters. You can serve cached pages to bots through your CDN. Some CDNs also let you rate limit specific user agents.
Before you block anything for performance reasons, check whether the load comes from real AI crawlers or from scrapers pretending to be them. Fake user agents are common. Matching request IPs against the ranges each AI company publishes is the reliable way to tell them apart.
Reading AI crawler activity in your logs
Log analysis tells you more than whether a crawler showed up. Look at which sections it visits most, how often it returns to updated pages, and whether it reaches your newest content. A training crawler that only sees your old pages will not learn about your latest products. A search crawler that never reaches a key landing page cannot help that page appear in AI answers.
If important pages get few visits from search-related AI agents, improve internal links to them, add them to your sitemap, and make sure they load fast without heavy scripts. The same habits that help Googlebot help AI crawlers too.
How to check if GPTBot visits your site
- Search your server access logs for “GPTBot” in the user agent field.
- Compare the request IP addresses against the ranges OpenAI publishes for this crawler to rule out impostors.
- Note which URLs it requests and how often.
- Check your CDN or firewall dashboard. Services such as Cloudflare label known AI crawlers and can show their traffic.
- Do the same for OAI-SearchBot and ChatGPT-User, so you see the full picture of OpenAI traffic.
If the crawler is not visiting at all, make sure you are not blocking it by accident through a wildcard rule, a firewall setting or a bot-protection service.
Common mistakes with OpenAI crawler rules
- Blocking GPTBot and assuming you have blocked ChatGPT entirely. OAI-SearchBot and ChatGPT-User are separate.
- Blocking OAI-SearchBot by accident with a broad rule, then wondering why you never appear in ChatGPT search.
- Using a wildcard “User-agent: *” block that catches every crawler, including search engines.
- Copying an old robots.txt template that lists outdated or misspelled user agent names.
- Relying on robots.txt to protect sensitive content. Robots.txt is a request, not a lock. Use authentication for anything private.
- Forgetting that a firewall or CDN rule can block bots even when robots.txt allows them.
GPTBot and other AI crawlers
OpenAI is not the only company with AI crawlers. A full AI crawler policy usually covers several agents:
| Company | Training crawler | Search or user fetch agents |
|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot, ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot, Claude-User |
| Google-Extended (a control token, not a separate crawler) | Googlebot powers Search and AI Overviews | |
| Perplexity | None stated for training | PerplexityBot, Perplexity-User |
| Common Crawl | CCBot, whose data many AI labs use | Not applicable |
Names and behavior change, so check each company’s documentation. Google-Extended is a special case: blocking it limits use of your content for Gemini training, but it does not remove you from Google Search or AI Overviews, which run on Googlebot.

A sample robots.txt for AI crawlers
This example allows search-related AI agents and blocks training crawlers. Adjust it to your own policy:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
If you want maximum AI visibility instead, allow all of them, including the training crawlers. Either way, keep your sitemap listed so crawlers find your important pages.
GPTBot and AI visibility strategy
Crawler rules are one small part of how AI tools see your brand. ChatGPT’s answers also draw on what other sites say about you: reviews, news, comparison articles and community discussions such as Reddit threads. A brand that blocks GPTBot can still show up in ChatGPT answers through those sources, and a brand that allows every crawler can still be missing if nobody talks about it.
So treat robots.txt as housekeeping. The bigger levers are clear, citable content on your own site and a strong presence in the places answer engines read. Our post on where ChatGPT gets its information explains those sources, and our guide to llms.txt covers another file some sites add for AI tools.
What OpenAI says about the content it collects
OpenAI’s documentation describes filters on what its training crawler keeps. It says it aims to exclude pages behind paywalls, sources known to collect personal information, and content that breaks its policies. Those filters are OpenAI’s own description, so treat them as a statement of intent. If you host anything sensitive, protect it with authentication rather than relying on a crawler’s filters or on robots.txt.
It is also worth separating crawling from use. Allowing a crawler means your public pages may be collected. It does not mean every page will be used, or that a model will quote you. Models learn general patterns and facts from many sources, and specific wording from one page rarely survives into an answer.
Legal and licensing context
The rules around AI training and copyright are still being argued in courts and legislatures in several countries, and some publishers have sued AI companies while others have signed content deals. Robots.txt rules are the main practical tool most site owners have today, and the major AI companies say they honor them for their training and search crawlers.
If your content has significant commercial value, talk to a lawyer about your terms of use and any licensing options. For most small and mid-sized businesses, a clear robots.txt policy and a quick yearly review are enough.
Checklist for managing AI crawlers
- Decide your policy for training crawlers and for search crawlers separately.
- Write explicit rules for each AI agent you care about in robots.txt.
- Check that your CDN, firewall and bot-protection settings match that policy.
- Verify crawler traffic against published IP ranges.
- Keep your sitemap current and listed in robots.txt.
- Review vendor documentation every few months for new agents and changes.
- Track whether your brand appears in AI answers, since that is the outcome that matters.
How to test your robots.txt rules
- Open yourdomain.com/robots.txt in a browser and confirm the file loads.
- Check the spelling and capitalization of each user agent token against the vendor’s docs.
- Use a robots.txt testing tool to check whether a given URL is allowed for “GPTBot” and “OAI-SearchBot”.
- Watch your server logs over the next few days to see whether crawler behavior changes.
- Recheck after site migrations, CMS updates or CDN changes, which sometimes overwrite robots.txt.
GPTBot FAQ
What is GPTBot?
GPTBot is OpenAI’s web crawler that collects publicly available content that may be used to train OpenAI’s models. It respects robots.txt.
Does blocking GPTBot remove my site from ChatGPT?
No. Blocking it stops training crawls. ChatGPT search uses OAI-SearchBot, which you control separately. ChatGPT may also fetch pages through ChatGPT-User when a person asks it to.
Does GPTBot respect robots.txt?
Yes. OpenAI says it follows robots.txt rules, and OpenAI publishes its IP ranges so you can verify real traffic from it.
Will blocking GPTBot remove my content from existing models?
No. Blocking affects future crawling. Content already collected before you added the rule is not removed from models that were already trained.
Is GPTBot bad for SEO?
No. It has no effect on your Google rankings. It only affects whether OpenAI may use your content for training.
How often does GPTBot crawl?
It varies by site. Check your server logs to see how often it visits and which pages it requests.
Should I block GPTBot on a business website?
Most marketing and business sites allow it, since broad awareness helps them. Publishers whose content is their product often block it. Decide based on how you earn money from your content.
Want to see where your customers are already asking? Get a free Reddit Opportunity Report showing which Reddit threads already rank for your keywords.