Field notesTechnicalCrawlers
robots.txt for AI crawlers: retrieval, training, and the difference that matters
AI crawlers split into two jobs with very different consequences. Blocking the wrong one removes you from answers; blocking neither may not be what you intended either.
AI Detective7 min read
The most consequential robots.txt mistakes we see are not typos. They are policies that made sense when written, applied to crawlers that do something different from what the author assumed.
Two jobs, often confused
AI-related crawlers broadly do one of two things, and the distinction determines what blocking them costs you.
| Purpose | What it does | Cost of blocking |
|---|---|---|
| Retrieval | Fetches your page so it can be used in answering a question, usually with attribution | Your content cannot be drawn on for those answers. This is the expensive one. |
| Training | Collects content that may inform future model weights | Your content does not contribute to training. Whether this matters is an editorial and commercial judgement. |
A publisher who blocks training crawlers to protect their archive, and inadvertently blocks retrieval crawlers alongside them, has removed themselves from answers while gaining nothing they wanted. This is a common and quiet failure.
The wildcard trap
The most frequent version of this problem is a broad rule written for a different era:
User-agent: * / Disallow: /
A staging-site directive that shipped to production
It is worth checking, because the consequences are total and the fix is trivial. Sites carry these for years without noticing, usually because they were correct on the day they were written.
Specific rules beat the wildcard group
Under the robots exclusion protocol, a crawler obeys the most specific group naming it, and ignores the wildcard group entirely if such a group exists. This has a consequence people rarely expect: adding a named group can accidentally remove protections you thought applied.
A worked example
Suppose a site wants everything public readable, keeps an admin area and an internal search results path out of scope, and wants retrieval crawlers explicitly welcomed. The naive version looks like this:
User-agent: * / Disallow: /admin/ / Disallow: /search/ // User-agent: OAI-SearchBot / Allow: /
Broken: the named group no longer excludes /admin/ or /search/
The intent was to add a permission. The effect was to replace the wildcard rules entirely for that crawler, which is now free to request the admin paths. The corrected version repeats the exclusions:
User-agent: OAI-SearchBot / Allow: / / Disallow: /admin/ / Disallow: /search/
Correct: exclusions restated inside the named group
The same applies to every named group you add. There is no inheritance to rely on.
What is usually worth excluding
Beyond anything genuinely private, a few categories consume crawl attention while offering nothing to read:
- Internal search result pages, which generate effectively unlimited URLs from query parameters
- Faceted filter combinations on catalogue pages, for the same reason
- Cart, checkout, and account pages, which are per-visitor and meaningless out of session
- API endpoints, which return data rather than documents
- Staging or preview paths that happen to be publicly reachable
Note that excluding a path from crawling is not the same as keeping it out of results. A URL that is disallowed but widely linked can still be known about; it simply cannot be read. If a page must not appear at all, that is a job for authentication or a noindex directive on a page that is allowed to be fetched — a disallowed page cannot be read, so its noindex cannot be seen either.
Testing what you wrote
- Request /robots.txt directly and read what the server actually returns, rather than the file you believe you deployed.
- Confirm it returns HTTP 200. A 404 means no policy is stated at all; a 500 may be treated as a blanket disallow by some crawlers.
- Check it is served as plain text, not HTML. Single-page applications sometimes return the app shell for unmatched paths, which silently swallows the file.
- For each named group, read it in isolation and ask what that crawler is permitted to do, ignoring every other group.
- Re-check after any migration or replatform.
A workable policy
- Decide separately what you want for retrieval and for training. They are different questions with different answers.
- State retrieval policy explicitly rather than leaving it to defaults, so behaviour reflects intent rather than inheritance.
- Repeat your disallow rules inside each named group.
- Keep the sitemap directive present and accurate.
- Re-read the file whenever the site is rebuilt or migrated, which is when stale staging rules escape.
What robots.txt cannot do
Two limits worth internalising. First, robots.txt is a request, not an access control. Well-behaved crawlers honour it; it is not a security mechanism, and anything genuinely sensitive needs authentication rather than a directive.
Second, allowing a crawler does not cause anything to happen. It removes one specific barrier. Whether your content is then retrieved, used, or attributed depends on everything else — whether the page is reachable, parseable, identifiable, and worth drawing on. Allowing access is necessary and nowhere near sufficient.