Separate AI crawlers, access, and cited sources

Explanation

OpenAI’s documentation distinguishes the purposes of these user agents:

• OAI-SearchBot is used to crawl for appearance in ChatGPT search features.
• GPTBot crawls content that may be used to train models. This setting is independent of OAI-SearchBot.
• ChatGPT-User represents a request initiated by a user action in ChatGPT/Custom GPTs; it is not an automated crawler or a signal for controlling appearance in Search. Robots.txt rules may not apply to it.

A User-Agent string can be spoofed and does not prove that a request came from an OpenAI crawler. To check requests, use access logs with the source IP and compare it with the current official searchbot.json ranges or a verified-bot signal from your CDN/WAF. Record response status, URL, time, and the protection system’s decision. Do not change WAF rules based only on a matching User-Agent. Allowed access makes a page potentially accessible to the system but does not guarantee selection or citation.

Fictional example

A synthetic access log for the training page /robots contains requests with the OAI-SearchBot User-Agent, but their IPs do not match the published ranges. This is an unverified crawler: record “claims OAI-SearchBot User-Agent; source unverified” and do not allow access on that basis. Later, a request from an IP in the current official range receives 403 from the WAF. That is evidence that a verified network source was blocked at that time; the owner may consider the rule separately.

The synthetic .test domain cannot be used for a real ChatGPT Search check. Run a query only if you have your own publicly accessible test domain the system can fetch. If no such check is possible, the product is unavailable, or no query was run, record “not checked,” not “no source.” Only a completed query with its response saved can establish whether the URL was cited in that specific result.

When using real access logs, retain source IPs only as long as the task requires and follow the applicable privacy and retention policy.

Practice

1. Read the platform’s official documentation and identify which crawler corresponds to its search feature.
2. In the fictional log, compare the User-Agent with the source IP and the current official range or your CDN/WAF verified-bot signal. Record the HTTP status and protection decision. Do not send a request with an impersonated User-Agent and treat it as proof of a crawler.
3. For a live check, use only your own publicly accessible test domain. If you do not have one or AI Search is unavailable, record “not checked.” If a check is possible, ask three neutral questions in the specified language and region; save the query, date, mode, full response, and source URLs.
4. Do not infer persistent visibility from one response.

Checklist

• OAI-SearchBot, GPTBot, and ChatGPT-User are not conflated.
• User-Agent is not used as authentication.
• Source IP is checked against the current official list or a CDN verified-bot signal.
• Site access is separate from whether a citation appears in an answer.
• Unavailable search is recorded as “not checked.”

Common mistake

Concluding “we allowed GPTBot, so the site is cited in ChatGPT,” or treating every OAI-SearchBot string in a log as genuine. OpenAI names OAI-SearchBot for search crawling; network requests require IP/platform verification, and citations must be checked in actual responses.

Primary sources

• Official overview of OpenAI crawlers
• Current OpenAI SearchBot IP ranges
• Searching the web with ChatGPT: site access requirements