ai-crawler-verify

ai-crawler-verify

A user-agent is a claim, not evidence. Anyone can send Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot). The operators that mean it publish a way to check — an IP-range file, or a forward-confirmed reverse-DNS suffix. This package runs that check.

from ai_crawler_verify import verify

verify("132.196.86.1", ua="Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)")
# {'verified': True,
#  'method': 'published-ranges',
#  'operator': 'OpenAI',
#  'matched_prefix': '132.196.86.0/24',
#  'reason': "address is inside 132.196.86.0/24, published by OpenAI at https://openai.com/gptbot.json", ...}

verify("203.0.113.9", ua="GPTBot/1.2")
# {'verified': False, 'method': 'published-ranges',
#  'reason': 'address is in no prefix OpenAI publishes for GPTBot', ...}

Three verdicts, and they are deliberately different things:

That third row is the point. Of the 150 crawlers in the table, 35 are verifiable from published IP ranges and 24 by reverse DNS; for the rest there is nothing to check, and a package that returned False for them would be lying about what it knows.

Install

pip install ai-crawler-verify

Python >= 3.8. Zero dependencies, standard library only. No network at import: 3052 prefixes from 15 operator lists ship inside the wheel (107.0 KB), snapshot taken 2026-09-02T14:41:10+00:00.

Command line

ai-crawler-verify 132.196.86.1 --ua "GPTBot/1.2"   # JSON verdict on stdout
ai-crawler-verify 66.249.66.1 --ua "Googlebot/2.1"   # reverse DNS, forward-confirmed
ai-crawler-verify --sources                          # which operator lists are bundled
ai-crawler-verify <ip> --no-dns                      # never touch DNS: fully offline
ai-crawler-verify <ip> --refresh                     # fetch today's prefixes first

Exit status: 0 verified, 1 not verified, 2 no verdict possible, 3 usage. That makes it usable straight from a log pipeline or a fail2ban action.

API

allow_dns=False guarantees the call makes no DNS query at all, which is what you want inside a request handler; the range check is pure CPU either way.

Why the prefixes are mirrored

Operators publish their ranges at a dozen different URLs in three different shapes, some of them CORS-blocked and some of them slow. The AI Crawler Index re-reads every one of them every six hours and publishes the union in one document — that is what refresh() fetches and what this snapshot was cut from. Every list is also linked back to the operator's own endpoint in sources(), so you can check the mirror against the source.

• The index: https://www.pathwren.workers.dev/c/pypi-verify/
• The prefix union this package uses: https://www.pathwren.workers.dev/c/pypi-verify/ip-ranges/all.json
• Companion package, offline user-agent classification: ai-crawler-index

Limits, stated plainly

• A True means this address is in the list that operator publishes. If an operator's list is stale or wrong, so is the answer.
• Reverse DNS is only as good as the hostname suffix the operator documents. Where this package has no documented suffix it reports the resolved hostname and returns None rather than guessing.
• Prefix lists rotate. The bundled snapshot is a snapshot: for a long-running service, call refresh() (it caches for six hours) or re-install.
• Data: CC0-1.0. Code: MIT.

Independent, non-commercial, run by automation. Not affiliated with OpenAI, Google, Microsoft, Apple, Perplexity, DuckDuckGo or any other operator named in the table; every prefix list belongs to the operator that publishes it.