ai-crawler-robots

ai-crawler-robots

A robots.txt that looks right and blocks nothing is the normal failure. This package reads the file you already have and tells you which of the 150 AI crawlers it actually stops.

from ai_crawler_robots import lint, audit, is_allowed

lint(open("robots.txt").read())
# [{'level': 'error', 'line': 3, 'code': 'ua-is-not-a-token',
#   'message': 'User-agent takes a product token, not a full user-agent string. '
#              'This group matches no crawler.',
#   'found': 'Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)',
#   'fix': 'User-agent: GPTBot'}]

is_allowed(text, "GPTBot", "/blog/post")
# {'allowed': False, 'rule': 'Disallow: /blog', 'line': 8, 'group': 'GPTBot', ...}

audit(text)["blocked"]        # ['anthropic-claudebot', 'openai-gptbot', ...]
audit(text)["unenforceable"]  # crawlers that will not obey it whatever it says

Install

pip install ai-crawler-robots

Python >= 3.8. Zero dependencies, standard library only. No network at import: 150 crawler records and 8 ready-made stances ship inside the wheel (169.6 KB), snapshot taken 2026-09-14.

Nothing in this package fetches a URL. There is no argument anywhere that takes one. refresh() is the only call that touches the network, you have to name it, and it fetches exactly one document: the same data.json that ships in the wheel.

The faults it finds

The most common robots.txt failure is not a syntax error — it is a file that a person reads as correct and a parser reads as empty. 23 checks, of which these are the ones that cost real traffic:

Every finding carries a line number, what was found, and a fix string you can paste.

Command line

ai-crawler-robots lint robots.txt                       # the faults, worst first
ai-crawler-robots audit robots.txt                      # every AI crawler: blocked / allowed / unenforceable
ai-crawler-robots check robots.txt --agent GPTBot --path /blog/
ai-crawler-robots diff old.txt new.txt                  # what changed by effect, not by text
ai-crawler-robots policy block-ai-training              # print a ready-made stance
ai-crawler-robots policy block-ai-training --merge robots.txt
ai-crawler-robots explain GPTBot                        # who it is, what blocking costs

lint exits 1 when it finds an error, 0 otherwise, so it works as a CI gate:

ai-crawler-robots lint robots.txt --strict || exit 1

audit — the question people actually have

$ ai-crawler-robots audit robots.txt
robots.txt: 150 crawlers, 12 blocked, 39 allowed, 5 unenforceable

BLOCKED (12)
  openai-gptbot            GPTBot              OpenAI       ai-training
  anthropic-claudebot      ClaudeBot           Anthropic    ai-training
  ...
ALLOWED, AND YOU PROBABLY MEANT TO BLOCK (7)
  google-extended          Google-Extended     Google       ai-training
  ...
UNENFORCEABLE BY robots.txt (5)
  These fetch on behalf of a user in a live session. Their operators state
  robots.txt does not govern them. Blocking needs an edge rule.

The third group is the one that surprises people. A user-triggered fetcher is not a crawler, most operators say so in their own documentation, and a Disallow aimed at one is a line that will never fire.

diff — did that edit do anything?

$ ai-crawler-robots diff robots.old robots.new
2 crawler(s) changed verdict for /
  google-extended     allowed -> blocked
  meta-externalagent  allowed -> blocked
44 unchanged. Formatting-only changes are not reported.

Compares by effect. Reordering groups, changing whitespace and rewriting comments produce an empty diff, because none of them change what a crawler does.

Ready-made stances

8 of them, the same bodies published at the index:

• allow-all — Every crawler on this index is named and allowed. Use when you want maximum reach into search and assistants and have nothing to withhold
• block-ai-training — Refuse the crawlers that feed model training. Keep the ones that put you in ChatGPT, Claude, Perplexity and Gemini answers
• block-all-ai — Training, AI search, user-triggered fetches and corpus builders, all refused. Classic search engines still allowed
• block-datasets — Refuse the crawlers whose output is a dataset other people train on: Common Crawl, AI2, Webz.io, Diffbot, ImagesiftBot
• allow-ai-search-only — Be findable and citable in assistants without contributing to training corpora
• block-seo-tools — Ahrefs, Semrush and friends. No user-facing consequence, and often the largest single slice of your bot traffic
• block-disputed — The ones repeatedly reported as ignoring robots.txt. Included for completeness — expect to enforce this at the edge instead
• maximum-ai-visibility — Allow every AI crawler and every search engine; refuse only SEO scrapers. For sites whose goal is to be found and cited by machines

--merge adds a stance to an existing file without touching the groups already in it, so your Googlebot rules survive.

Where the data comes from

The AI Crawler Index — an independent, non-commercial public reference. Every crawler record links to the operator's own documentation, and explain prints that link. Presence in the table means the operator documents that crawler; it is not an opinion about whether you should allow it.

refresh() fetches today's table and caches it six hours, which is how often the index behind it is rebuilt:

from ai_crawler_robots import refresh
refresh()   # {'crawlers': 150, 'generated_at': '...'}

Companion packages

Same table, different questions:

• ai-crawler-index — what does this user-agent claim to be?
• ai-crawler-verify — is that claim true? Checks the address against the operator's published ranges.
• ai-crawler-logs — who was actually in your access log, and what to paste to act on it.

Licence

Code MIT. Bundled data CC0-1.0. Independent and non-commercial; not affiliated with, endorsed by or speaking for any crawler operator named in the table.