ai-crawler-logs

ai-crawler-logs

Who was actually in your access log? Paste the log in whatever shape your stack emits it. Every crawler gets identified and attributed to its operator, rolled up by category, and turned into a robots.txt or an edge rule for the traffic you really received.

from ai_crawler_logs import triage

report = triage(open("access.log").read())

report["summary"]["by_operator"]   # {'OpenAI': 412, 'Anthropic': 88, 'Google': 61, ...}
report["summary"]["by_category"]   # {'ai-training': 500, 'ai-search': 91, ...}
report["summary"]["ai_share"]      # 0.4231
report["unmatched"]                # the strings nothing in the table explains

Install

pip install ai-crawler-logs

Python >= 3.8. Zero dependencies, standard library only. No network at import, no DNS ever: 150 crawler records and 8 ready-made robots.txt stances ship inside the wheel (169.6 KB), snapshot taken 2026-09-05.

It reads the log you have

Formats are detected per line, so a mixed paste works and every row reports how it was read:

The everyday path is one shell line and no log parsing at all:

awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | ai-crawler-logs -

Command line

ai-crawler-logs access.log                    # who was in it
ai-crawler-logs access.log --ai-only          # only training/search/fetch/dataset crawlers
ai-crawler-logs access.log --json             # the whole report, one row per line
ai-crawler-logs access.log --robots block-ai-training
ai-crawler-logs access.log --waf nginx        # apache | caddy | cloudflare | haproxy
ai-crawler-logs access.log --verify           # needs ai-crawler-verify installed
ai-crawler-logs --stances                     # the 8 ready-made stances

$ ai-crawler-logs access.log
1284 request(s), 743 identified (41.2% AI), 96 distinct address(es)
formats: combined=1284

crawler                        operator          category        hits
GPTBot                         OpenAI            ai-training      412
ClaudeBot                      Anthropic         ai-training       88
PerplexityBot                  Perplexity        ai-search         61
...

not in the table (23 distinct):
   1204  Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537...  browser-shaped string
     31  python-requests/2.31.0                                        python-requests

The output you can paste

$ ai-crawler-logs access.log --robots block-ai-training
# Generated by ai-crawler-logs 1.0.1 from an access log.
# Stance: block-ai-training — ...
# Only the 6 crawler(s) that appeared in the log are named.

User-agent: GPTBot
Disallow: /
...

Only what was actually there. A robots.txt naming 150 tokens you have never received is a file nobody will ever maintain; pass --all if you want the whole stance anyway.

For the fetchers that will not be stopped by robots.txt at all, --waf writes the edge rule instead:

$ ai-crawler-logs access.log --waf nginx
map $http_user_agent $ai_crawler {
    default 0;
    ~*(GPTBot|ClaudeBot|PerplexityBot) 1;
}
if ($ai_crawler) { return 403; }

A user-agent is a claim

Everything above is a claim matched against a table. Nothing in it is proof that the client is who it says — the string is free to type, and the busiest strings in most logs are forged. With ai-crawler-verify installed the rows that carried an address get checked against the operator's own published ranges:

pip install ai-crawler-logs[verify]

CLAIM CONTRADICTED BY ADDRESS (3):
  line 88    203.0.113.9      claims GPTBot — address is in no prefix OpenAI publishes

That list is the only thing here that rests on evidence, and it can still be wrong two ways: a mirrored prefix list is up to six hours stale, and some operators route through addresses they do not publish. It is a reason to look, not a verdict about a person.

Where the data comes from

The AI Crawler Index — an independent, non-commercial public reference. Every crawler record is checked against its operator's own published documentation. refresh() fetches today's table from https://www.pathwren.workers.dev/c/pypi-registry/ai-crawler-logs/data.json and caches it six hours; it is the only network call in the package and you have to name it.

Companion packages

• ai-crawler-index — what does this user-agent claim to be?
• ai-crawler-verify — is that claim true?
• ai-crawler-robots — does your robots.txt block the crawlers you think it blocks?

Licence

Code MIT. Bundled data CC0-1.0. Independent and non-commercial; not affiliated with, endorsed by or speaking for any crawler operator named in the table.