agentpulse-asgi

agentpulse-asgi

AgentPulse for Python: detect AI agents, serve them clean Markdown and declare Content Signals in FastAPI, Starlette or any ASGI app. Standard library only.

pip install agentpulse-asgi

Usage

from fastapi import FastAPI
from fastapi.responses import PlainTextResponse
from agentpulse_asgi import AgentPulseMiddleware, ContentSignals, content_signals_robots_txt

signals = ContentSignals(search=True, ai_input=True, ai_train=False)

app = FastAPI()
app.add_middleware(AgentPulseMiddleware, content_signals=signals, on_event=print)


@app.get("/robots.txt", response_class=PlainTextResponse)
def robots() -> str:
    return content_signals_robots_txt(signals)

Requests that prefer text/markdown (RFC 9110 q-values) get the HTML page converted to Markdown; everyone else gets the original HTML. HTML responses carry Vary: Accept. Markdown responses carry x-markdown-tokens / x-original-tokens estimates (Cloudflare's Markdown for Agents stopped sending them on 2026-10-09) and, as Cloudflare does, no ETag/Last-Modified, since conditional requests cannot be honored for a converted body.

What it does

• Agent detection from 22 operator-documented User-Agents (OpenAI, Anthropic, Perplexity, Google, Meta, Mistral, Apple, Amazon, ByteDance, Common Crawl). The catalog is generated from the same source as the TypeScript packages and both pass the same conformance vectors.
• Optional IP verification (AgentVerifier + get_client_ip=client_ip_from_scope): checks the client IP against the ranges each operator publishes → verified, mismatch (likely spoofed) or declared. Lists download in the background; failures fall back to declared.
• Safe HTML → Markdown: no regex over HTML, scripts/styles/comments/hidden content dropped, only http(s)/mailto/relative URLs kept, nesting capped at 512.
• Metadata: a YAML frontmatter (title, description, image from <meta>, with name taking precedence over Open Graph) whose values are JSON-quoted strings that cannot inject keys, and the page's valid JSON-LD appended as one fenced JSON value (an array when there are several blocks; malformed blocks and blocks with javascript:/vbscript:/data: strings are dropped; 64 KB total).
• Resource limits: at most max_body_bytes (2 MB) read per response; the app is told to stop (http.disconnect) once enough has been read; conversion runs in a worker thread; a conversion cache and per-client/global rate limits protect the CPU (rate-limited requests get the original HTML).
• Telemetry: on_event (sync or async) runs in the background with the path and agent — never the client IP or the query string. IngestClient(write_key) sends events to the AgentPulse API (app.add_middleware(AgentPulseMiddleware, on_event=telemetry.on_event), then await telemetry.shutdown() in your lifespan). It keeps a bounded queue (500 events; the oldest are dropped and counted), sends batches every 10 s or at 50 events, and retries with backoff and Retry-After. Paths are redacted (emails, UUIDs, long numbers, hex and tokens) before they leave your server. Plain human requests are sent only as a counter.

Requirements and security notes

• Python ≥ 3.10 with the 2025 html.parser security fixes (CVE-2025-6069, gh-102555, gh-135661). Older patch releases leak unclosed HTML comments as text and parse some inputs in quadratic time; the middleware checks the interpreter at startup and refuses to run on a vulnerable one (UnsafeHtmlParserError).
• asyncio servers (uvicorn, hypercorn).
• Behind a reverse proxy, let the server set the client address from trusted proxies only (uvicorn --proxy-headers --forwarded-allow-ips=...); never read X-Forwarded-For yourself.
• Shared caches/CDNs that ignore Vary could serve Markdown to browsers; configure them to vary on Accept or not to cache negotiated pages.

License

MIT