GPTBot, OAI-SearchBot, and what blocking each does
OpenAI is the only one of these companies running a visible split between a training crawler and a search crawler. Treating them as one decision is how sites end up excluded from a surface they were paying to appear in.
Checkable · robots.txt plus your server logs
Two agents, two jobs
- GPTBot
- The training crawler. Blocking it is a decision about whether your material contributes to future models. It is a legitimate choice and it is about the long term.
- OAI-SearchBot
- Associated with surfacing live results. Blocking this one is a decision about whether you can be found and cited now, which is a different question with a much shorter feedback loop.
A blanket disallow covers both, and that is usually what is actually in place when a contractor believes they made a considered choice. Plugins and previous developers add these, and nobody rereads the file.
What blocking actually costs you
Blocking the search crawler removes your pages from the pool the live half of the system can read. It does not remove mentions of you from elsewhere on the web, which is the part contractors find counterintuitive: an engine can still describe your business from a directory listing, a review platform or a forum thread, while being unable to read the site where your actual details are correct.
That is the worst of both outcomes. You are describable and not correctable.
How to check, properly
Two things, and the second is the one people skip. Read your robots.txt and note what each agent is allowed. Then grep your server access logs for the user agent strings and see what has actually visited and how often. The file states your intent; the logs record what happened, and they occasionally disagree.
A contractor with a correctly permissive robots.txt and zero crawler visits in ninety days has a different problem from one whose file says disallow, and only the logs tell them apart.