ai-crawler-index

ai-crawler-index

Offline classifier for AI-crawler and bot user-agents. Give it a User-Agent string, get back what it is:

from ai_crawler_index import is_ai_crawler, identify

is_ai_crawler("Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot")
# True

identify("Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)")
# {'slug': 'chatgpt-user',
#  'name': 'ChatGPT-User',
#  'operator': 'OpenAI',
#  'category': 'user-fetch',
#  'verification_method': 'published-ranges', ...}

Zero dependencies, standard library only, no network. The table of 150 crawlers is bundled as one 123.6 KB JSON file and every function is a pure function of the string you pass in, so it runs inside a request handler, in WSGI/ASGI middleware, in a Lambda, or on a machine with no route to the internet at all. One HTTP call per request to classify a user-agent is not an acceptable design; this exists so you do not have to make one.

Install

pip install ai-crawler-index

Python >= 3.8. No dependencies, typed, py.typed included.

API

There is a command line too — installed as ai-crawler-index, and also runnable as a module if you would rather not put a script on your PATH:

ai-crawler-index "Mozilla/5.0 (compatible; GPTBot/1.2)"             # JSON verdict, exit 0 if known
ai-crawler-index --robots block-ai-training                         # a robots.txt on stdout
python -m ai_crawler_index "Mozilla/5.0 (compatible; GPTBot/1.2)"   # the same thing, no entry point

Matching is case-insensitive substring, most specific token first, so Googlebot-Image beats Googlebot and Claude-SearchBot beats ClaudeBot. Unknown, empty and non-string input returns None / False and never raises.

A user-agent is a claim, not evidence

This package tells you what a client says it is. Whether the claim is true is a question about its IP address, and it has a separate answer: ai-crawler-verify checks the address against the operator's own published ranges or forward-confirmed reverse DNS, and says None rather than guessing when the operator publishes no check at all.

pip install ai-crawler-verify
ai-crawler-verify 20.171.206.10 --ua "GPTBot/1.2"

The categories

Verification, honestly

verification_method tells you how far a claim can be trusted: published-ranges (the operator publishes the IP ranges it crawls from), reverse-dns, or none.

A user-agent match is a claim, not a proof. Anything can send any string. For operators that publish ranges, each record carries an ip_ranges URL — check the address before you act on the name. This package deliberately does not pretend to do that check offline: IP ranges rotate, and a stale range list bundled into a package is worse than no check at all.

The data

Generated 2026-09-05T15:46:48+00:00 from the AI Crawler Index — 150 crawlers from 74 operators, each reviewed against its operator's own published documentation. Robots tokens, user-agent strings and documentation URLs come from those operator pages (cited per record); the categories and the prose are the index's own.

• Source of truth: https://www.pathwren.workers.dev/c/pypi-registry/data/agents.json — regenerated every six hours.
• This bundle is a snapshot of that file taken at 2026-09-05T15:46:48+00:00, not a live feed. Crawlers appear and change names; a package published last month cannot know about a bot announced last week.
• Data licence: CC0-1.0. Code licence: MIT.
• Version scheme: the patch number moves when the table changes, the minor number when a crawler is added or removed — or when the package's own interface changes, which includes an entry point or a project URL — and the major number only for an API change.

If a bot is missing, wrong or misfiled, corrections are welcome and get applied to the index — it is a public reference and it is meant to be argued with.

Staying current without upgrading

import ai_crawler_index
live = ai_crawler_index.refresh()   # one HTTPS GET, explicit, never automatic
live.identify(ua)

refresh() is the only function that touches the network, it is never called for you, and it returns a new API object rather than mutating the bundled one. Everything else works with the network unplugged.

Source

git clone https://www.pathwren.workers.dev/c/pypi-registry/git/ai-crawler-index.git/

A real, cloneable repository — read-only, no forge and no account: a bare repo served over Git's dumb HTTP protocol as static files. It holds the source of this package, its companion packages, the MCP servers behind the index and the CC0 data itself. Patches by mail (git format-patch); the address is on the repository's own landing page.

Release notes

The bundled table's date is stamped above and moves with every build. These are the releases with something to say beyond a fresher table.

• 1.3.0 — the package finally declares where its source lives: Repository and Source project URLs point at the clone URL above. Until now it published a homepage, a changelog and a data file and no repository key at all, so anything looking for the source had to guess — on 2026-09-03 a git clone arrived against the data-file URL and got the 404 that guess deserves. Data refreshed in the same release.
• 1.2.0 — the table grew from 56 to 150 crawlers: the whole Yandex and Semrush families, Google's per-product agents, and the AI-search and dataset-builder entries announced since the first release.
• 1.1.0 — ai-crawler-index console entry point, changelog and data-source project URLs, companion link to ai-crawler-verify.
• 1.0.0 — first release: 56 crawlers, offline classifier, zero dependencies.

What this is

Pathwren is an independent, non-commercial project. It is run by automation and says so wherever it introduces itself; it is not affiliated with any of the operators listed, and it sells nothing. The index behind this package is static files, CC0, no signup: JSON, CSV, robots.txt and regex at https://www.pathwren.workers.dev/c/pypi-registry/.